Welcome to day. So this is the first time garage is presenting at Boston. Usually it's
either a counter or a likes. This time it's me. I'm not part of the original developer group.
I'm actually more like a contributor and part of the Duffler Association, which is a shifurding
the garage project. So I can't answer deep chord questions, but I've been handling the
community channel for quite a while for people who know me. And I think it's interesting
for us to have a point a checkpoint today after five years of garage on the way we are
and what you can do with it. So let's get going. The fur just a small word on it. We are
a French association. So it's just a group of people. We try to promote digital sovereignty
and privacy. And of course that goes so hard to word self-hosting as an alternative to
big cloud providers. This means that we, because we do self-hosting, we need resiliency. And that
means that we get good at time and relatively low supervision and the low manpower that goes
with volunteer work. For anybody that needs to catch up, why is garage exactly? Why is it
different? Garage is a self-hosted drop-in replacement for the AWS S3 protocol. I mean,
AWS Object Storage General, but also there's a ton of product now that use that protocol.
It is a bit different because it's mainly implemented resiliency through geographical
resiliency. So the idea was to use not to put stuff in it at our center. We've ridden and power,
ridden and connectivity, and instead to use smaller machines in a location that do not have
redundancy, like re-appartment or like a shared location. And to implement the resiliency on
top of that, so meaning that you have multiple geographically copy of it out of, and then if one
location goes down, then you can just access it in another location. So you can use it to
host a website. We host a defler, more than 600 websites on garage for quite a while now.
You can store media, pictures, videos, as well. You can use it for peer cube as a backend.
And a ton of people are using it as well as a backup target. So classical S3 protocol usage.
Why is garage different? There's a ton of over-esprimplementation order.
Garage is a bit different because it's coordination free. Meaning that we don't have a central
elected master. There is no election going on. Every request, basically, we need to get consensus
from the cluster. And that makes it extremely resilient to latency. I'm going to talk a bit about
that later. But that's the main difference. We do not have a raft. We do not have elections.
And that also creates some constraints on how the clusters are managed. But the benefit is that
it's extremely beneficial for a situation where you have high latency between nodes.
Because the lack of elections will mean that basically every request is going to give you the
base latency. This is a quite old graph at this point. I didn't rerun the testing, but the point
still stands. Basically, here what is shows you is that when you operate on objects in garage,
when you're going to get latency, when you have high not-to-not-latency. We're talking about
a hundred milliseconds. This is not that a central latency. This is like almost Europe and US latency.
You get much better latency on some operations. And that's what really where
architecture is about. Because other solutions are much more optimized if you want to just
put stuff in Iraq and then have just like horizontal scaling that is very located physically speaking.
So the consistency model is not acid. We only have read after write consistency, which is still
better than eventual consistency. I think that's as well what's as free. Had initially another
originally. I think they improved a bit for models since then, but they still say that they
have we have to write consistency. It might be better. But yeah. We talked about geographical
deployments. Garage has a zone concept that allows you to spread your deployments across different
zones, which are geographical locations. And then you can have multiple nodes per location.
So you can have, for example, free locations. And then have two nodes in one location,
free nodes in another locations. I'll also do to scale a bit for cluster as well.
What makes Garage a bit different? So I already talked about it. It's really the geographical
distribution aspect. Here we can see an example that you could have with multiple zones. One in
the UK, one in France, one in Belgium, one in Germany, and one in Switzerland.
And you can see that the chunks of data are actually spread. So this cluster has a replication factor
set to free. And you can see that the blue chunk is actually replicated in Belgium in France and
in Switzerland. But the red chunk is actually replicated in Germany in France and in the UK.
So on top of the cluster, you have a replication factor. And then that replication factor would
basically govern all of the copies of that habit happens in the cluster itself.
This is a quite old compatibility list. We have more now. But these are some of the software
that you can find that are using S3 for storage. And that can be configured to use Garage.
Again, because some software have hard-coated dependency to Amazon S3. They are not always
compatible. But more and more software now are becoming compatible with self-hosted S3 alternative.
So this is a very exciting. You can find more at today's list on the website. We have a section
in here called the integrations where you can basically list all of the software has been tested.
We've configured some tools for them to help you get started.
So nothing was new at this point. All of this was from over Garage Talks.
Now this is the new content. This is just a short history. We have a roadmap that we had so far.
So the initial beta release was in 2021. We have the actual first release version that we
talked about to use in 2022. And then we had the first table release in 2024.
There was a nice talk done by Alex at first them. That was an onswing this.
The one has been here for quite a while. We decided I was talking with Alex today. We decided to
deprecate it this summer one year after we released the 2.0 version. Right now we're still maintaining it.
And then the 2.0 was released a couple of months ago. It actually doesn't have a ton of
breaking changes. But it has a brand new HTTP administration API, which is one of the reason why we
had to release a new major version. And then we changed the old replication mode setting to split it
between the replication factor and the consistency policy. So that made things a bit more configurable.
It's for you to pick your replication and your consistency factor differently.
We are thinking about releasing a free.0 version. We are really thinking about
versioning support. Garage does not support right now as free versioning, which is a huge issue
for a lot of software that relies on it. So there's a couple of things like that.
Versioning, tags on buckets, object retention policies. Those are things we are thinking about.
We are not making any promises, but they are or objectives for the freedom of the version.
Okay, so one thing I wanted to do for quite a while is to spend some time on best practices
for guard deployments. We have a ton of people asking your questions, so it's
sometimes a bit clunky. So I have a couple of sites of that. And then we can
either do a quick demo or I can take questions. So the things you should know, if you don't
know garage yet, we don't have to support within Garage. So Garage allows you to configure
a website in point. And that website in point is spinach dp. It is by design. We intend to
do your own certificate management and outside of the garage. And so it means that you have to
set up your own proxy. We have documentation for engineers, for caddy and a couple of hours.
Usually it's not really a big issue, but it's something that we made that just consistently back
back then. We don't have yet a free anonymous access, which goes for issue for some
softwares. Basically, we advise you to use the website in point for that, but not also for
match to use that way. So it's something that we're freaking about. I think there's an issue right
now about it. And I was discussing with Alex, we might have a PR coming in couple of months.
And it might require us to release any major version, we're not through yet.
Due to the lack of, like I said, like cluster management or elections, we don't have a way
to kind of like automatically configure the cluster. And so that rolls always fall back to the
operator. So that means that when you create a cluster, the nodes can discover themselves,
but they will not autoconfigure themselves, because the cluster layouts that you will configure,
which determines which nodes does what, is something that needs to be done by some from a
30, but we don't have. So this can be node. Sorry, this can be done by interacting with any nodes.
But it falls back basically on an operator to do that. And that operator can be either human,
you can do that manually with a CLI, or you can also build your own scripts or operator in the
Kubernetes, so that's the way to do that for you if you want. A couple of people in the
community have done that. And then you're going to make cluster set a bad way.
Verification factor is something that is defined at cluster creation, and that we
strongly advise not to change. They are in supported procedures to do that, but please don't try it.
We had issues in the past, and it's very annoying to recover cluster in that state. So basically
what we advise is really think about your replication factor first, and then if you need to change it,
just slip in a cluster and copy your data. Come on, pitfall. The default region in garage is
garage, not USD1. Very common issues. And then we had a discussion about access well yesterday,
and it was saying that really from his own experience as well, salvaging clusters from people
that may or else after disasters do not use the degraded consistency, but I see really only
it for that I recovery. Don't use it for like a regular cluster.
Well, I had a lot of questions along the last two years on what hardware people should use.
We really bit garage to tell her any hardware, but it works best on some hardware, at least.
Those things are pretty obvious. I think for most people in the storage community,
but they might be not necessarily obvious for people that just want an S3 storage store,
and they don't really know a lot about storage. So the first rule, when you start to your storage,
is to not put your storage back in on something like an FS or an S&B. It's a reciprocal disaster.
So please don't do that, especially from the Metadata folder. The base on S&B yarn and FS don't
tend to perform really well, and sometimes they have bizarre issues, especially when you cut the
network. So don't do that, please. Get a right-intensive flash disk for the Metadata folder.
To be honest, like anything flash should work for like very, very small use case,
but if you have any service use, and if you have a relatively big cluster,
especially if you have nodes that come and go, garage actually use the Metadata database
to store queues of blocks that needs to be received, and those queues can be very, very
right-intensive. I can tell you that at the third, we had normal SSDs, like some some, like
retail SSDs, and we wore them out in less than two years. They basically retot their,
their wear leveling. So try to find something a bit flash, sorry, like geared towards a
right-intensive workload, and keep an eye on the wear leveling of your drives.
With vise too, as well use red one for that, and to use a copy on the right-fast system,
to do stop plotting. You can also enable compression. You get a bit more latency, but it
basically compressively well, most of the base does. Get HDDs of flash for your data folder,
use XFS for your data folder, garage creates a lot of small files, so you need something that has
a lot of eye nodes. And here you have a choice. We have a multi HDD mode for the best performance,
so guys will basically take each drive independently, and then your application can be
basically geographical, like you don't need to have multiple rebukers in the same zone,
if you can retrieve them from another location. If you really don't want to think about it,
you can use red, and you kind of red below it, ZFS, or and DDA more or the best,
but you will leave a lot of performance on the table, because you're absolutely basically
building a TV array. But on the CPU and the RAM, we don't need that much, of course,
depends on the use case. If you have a cluster, we've one bit above data,
having a bit of RAM will help for metadata. Above metadata, we have multiple metadata
engine in garage. Basically, the metadata is free things that's all of the mapping between the
files and the blocks. It's also the bucket, the bucket, the access on all of that stuff.
But also, any file that you upload to the cluster, well, are below three kilobytes,
will actually be stored in the metadata and engine itself, and not on disk. So it's something
good to know if you store tons of small files, you might not see those blocks in the data
folder, there would be a metadata side. We had multiple engine, along with yours,
sled has been removed. After one dot zero, if you haven't done that, you need we need to migrate.
We have the tools for that. It's basically take your no-down, execute the migration, and then
put it back up. The two means that we have right now are SQLites, which is the safest one.
It is recommended for smart clusters. It has a couple of performance issues, but it works,
it's pretty rock solid in terms of power loss and that I have consistency.
And then anundibi is a bit faster than SQLite. It's there for big clusters. However, we had a
couple of plenty issues a long time ago, and if you have an unrollable power, then I strongly suggest
you use a UPS with anundibi over to backup frequently. Anundibi has also a limit for the key
size that has been hidden by a couple of people using backup services. So the maximum length
of a usable key is four hundred and eighty bytes with anundibi. So if you need long case,
you might have to use a SQLite. And then we had a newcomer, which is FJL, which is experimental
of this point, but it's already in the code base, so you can try it and let us know what you think.
The method that engine can be downloaded, like there is no really restrictions. You can have a
couple of nodes using a SQLite and a couple of nodes using anundibi. We really thought
garage to be something that was meant to be your distributed, but following the Minio people
turned off hard. It's great like that. We had a ton of people asking us for single load deployments,
which are a bit clunky, because it's really meant to be a cluster. So we, of course, if you have
a single load, you will lose resiliency. Make sure you have backup for a metadata. If you use
anundibi with single load and you have a poor cut, you might end up with a corrupted database,
and it's very annoying to recover from that. We really don't have a way to rebuild the metadata
from the data. So you will have to basically, you will potentially lose some data if you
did that. Like you will at least lose a lot of the file names and any file that might be stored.
So we advise you to use SQLite if you did that. And potentially put a UPS on top. Some people
really have super flaky power, and they are not really crushed like every two days, and the chances
for you to have a corruption with that is relatively high. Right now, the experience to set up
garage in a single load is a bit long. There's a ton of components that have that needs to be
done. Someone from Recommonity actually put together a bit of a wrapper for that. We might actually
integrate that into garage itself. So I was discussing with Alex yesterday and we were thinking
about integrating like a garage, serve, dash, dash, single load that would basically do a
single load setup. So that might come into following the move. With the node deployments,
Joe distributed zones tried to have at least free. Keeping mine your network and Io, we had people
built a very, very big clusters on top of DSL effectively, and then they lose a node and they
have to build it and it takes literally half a month. So be careful about that. You can easily
accumulate data and then keep an eye on how long it would take for you to recover from that.
Please set up monitoring if you don't have that, it's very hard to understand what's happening.
You can get some inside, but it's difficult. To give you a bit of an ID, Duffer has been running
a nine-terbat cluster, which is three nodes, three nodes and two nodes. So three locations,
replication set to three. Over regular fiber across Europe, the between Lyon and Burstall,
the bit of Britannia as well. We heard that there are tons of bigger clusters out here,
up to the petapites, but we don't really have a lot of data because we don't have any telemetry
in the project by design, so we don't know. Tell us if you have big clusters, we're interested.
I'm going to go a bit faster, even. I don't think we're going to have time for a demo.
But we, on deploying, really use whatever you want, garage is a single binary. We compile it,
we went to Great Land to compile it statically, so you can really just drop it on the system.
You can also use Docker. We have Kubernetes and console integrations for not discovery,
but you will have to set them alive manually or try to script on an operator to do that.
You can use something called Gateway nodes, which don't store anything, but can kind of
like geared your request toward the best node for load balancing. And if you find the default settings
or default missing time to be a bit conservative, you can adjust the missing tranquility,
which is basically how much time a garage sleeps between missing operations
to avoid overloading the cluster as a whole. There is a Kubernetes storage controller
that is in the work as well from the community, so you can take a look at that if you run into
Kubernetes. We have someone from the community that created a UI this year. It's already usable.
It's delivered to the admin API. So thanks to them. I forgot to put the name in here. I should
have actually. But it's not here. You can find it. And then we also have an official UI coming
later this year, but it will be embedded within the product itself. So those are mockups for now.
But this is a very actively being worked on things to some friends that you got from
an internet this year. So Valam, I want to do short paths on the garage metrics. So this is what
to get if you're a version of a default dashboard. Really, the two things that you need to be aware
are those two things here. So the racing queue length and the racing overall blocks. It will
still shut up. It means that you are issues because your clusters cannot sync blocks correctly
across the clusters. The rest are not necessarily an important but they give you more
insights on the information. One thing that a lot of people don't know is that garage for safety
reasons don't delete blocks right away if you delete a file. So if you delete a file in this
spray PI, a garage will actually wait on those 24 hours before actually removing the block from
storage. So if you create and remove a lot of files, then potentially you might see your storage
going up for a while because garage will not remove those yet. So the the table gcq length
has a bit of that as well. Like it was, show you the blocks as very remove black garage across time.
One thing is going wrong. Enable for debug logs. We had a ton of people with
authentication issues because of the S3 way of signing a request. So be careful about your
reverse proxy configuration. Check the tranquility if you think the racing's are slow. And then
LMDB has a bad tendency to kind of like being blown it up a time goes on. You can just shut down your
garage note, do an MDB copy with the C parameter to compact it and then it will go back to
normal size. We take questions on the garage matrix channel in English and you can log issues as well
on gith.daflow.fr, which is a 4G Joe instance. We also have a mirror of a code on github but it's
basically just here for the people that want it on github and we don't take issues there.
And then if you open an issue, if you have any problem, please provide the output of
garage status and then garage status as well. If you're moving from main to main, I don't think
I will have done to do a demo. But basically, the idea is that you need to list you
to break it on keys and then you create both buckets on the garage cluster. Right now, you will
have to create new keys because garage enforces that the keys start with the specific string.
We will put a PR soon to basically add a parameter to skip that and use garage key import to
basically import your own key and then you just loop over the buckets with our clone and then
you just copy for that up. I thought blog posts would actually be better for this as I don't have
time for a demo here. And that's it, I think. I can do some demo.
Does that include questions? Does that include questions?
Yeah, that's your whole smart. Okay, so that's five minute for questions.
Yeah, so the question is do you need gateway nodes within a cluster? The response is no. Any nodes
that can take request. Gateway nodes are only here for helping. Basically, they have a copy of
the metadata and the cluster and they are able to direct. So what might happen if you hit any nodes
is that you will bounce like the node might not have a copy of it at all and it will might go elsewhere.
And so you lose one RTT in latency. What you can do is you can run a gateway locally.
And then that way you can just talk to the gateway locally and then the gateway will basically
ban you directly to the correct nodes. So that's much faster in that regard.
One caveat though is that gateway nodes are fully part of the cluster itself.
And so they basically have full access to the entire cluster. So that means that
whatever if you're running a gateway locally, then you have to trust that that compute
to because it basically has full embedded access to the entire cluster. Yes?
And you may come that it's more than a consumer. I'm guessing if it's doing a single one
price, do you have any info on what's the block size for those price? So you can optimize
that best. Okay. So the question is, we're doing a lot of block rights. And synchronous
we're back rights here. How to optimize that? Especially if you have something like ZFS but
has a block size. So that's a good question. I think I don't know what LNDB used by default.
I would assume it's something that's 16K or similar. That's usually common for databases.
For the metadata, so that depends a bit on what you pick. For the data itself.
So garage has a block size setting. And it will effectively split any incoming file up to that
block size. So if you have a block size, it says to set to I think the default is one megabyte.
Any file that is below one megabyte will start in the single chunk. So it will appear as one file
on your data folder. And if you have a file that is above that, like then it will be split in
multiple chunks. We actually advise to raise that block size because there's no rental
consequences. The only consequences are the additional buffers because when garage
under the block, it's 101 by one. So the bigger you block, the more memory you need. And also
like within RP cycles because it's brought by block, then your RP cycles will last longer. But that's
more or less it. So big clusters really go up in in block size between sometimes like 10 megabytes
or even 15 megabytes. If you store a lot of big content, like people storing video content,
it can go very high. And that gives in a lot of more bandwidth for that.
So the question is what is the metadata folder used for? And it's really only used for that.
And a couple of static configuration for like the layout configuration as well.
So the metadata folder needs to be basically tuned for the metadata engine that you use.
You mentioned that what's currently directly deleted, where a file is deleted,
is possible to validate the file will be updated.
So the question is blocks are not necessarily deleted right away when you delete the file in
this free. Is it possible to recover that? Right now it don't think it is. I mean technically
the blocks are still on disk, but the metadata has been deleted. So I mean one thing though is
if you have a metadata task snapshot, then you would be able to restore that and then to just
give a block to garage and then we scan for blocks and it will just connect again.
So the question is did we consider implementing a recycle bin? This basically would be implemented by
SV versioning. Yes?
If you are implementing versioning, we also implement object or
block. So the question is if we implement versioning, we will we be also implementing
implementing object locking. I need to look into it. I think it's something that we haven't done yet,
but that's something that's required, why not require it, but is high in demand by a lot of backups
software that is it for a ransomware protection. So yeah. And the times up, you can find me,
I will be hanging around. You can find me a couple of questions.