Hi everybody, again, thank you for coming to my talk and I'm going to be talking about my
experiences trying to make asynchronous Iowa work on all of the BSTs and Solaris, Alumos
and actually 10 operating systems. Most of which didn't work but some of which gave me
a lot of ideas about work that could be done and I've done some experimentation and
I want to talk about that and see if I can make contact with other people who might be interested
in the same source of problems. So I actually do most of my post-paste development on a
3BSD box and I've made a few small patches to 3BSD and I was punished for the commit
bit for that but I haven't, I'm not a particularly prolific contributor to 3BSD. I would
like to do more work and some of this is experimentation in that direction. Okay, so I wonder
if I can change page down, sorry we're going to have to scroll. So the talk has cut
up into five sections. The first three have slides and the second one just sort of discussion.
So what databases want from the IOS stack? We can watch the giant world scroll past.
So firstly, if you go back to the origins of Postgres, it was started in 1986 at Berkeley
and that's the same time in place that BSD was happening. So a lot of the early work in Postgres
was done on and and other related projects was done on BSD systems and some various spin-off
systems, sun machines and dynamic machines and they were in that sort of general family.
And the people that worked on that wrote a lot of papers about how terrible the whole
file system interface of Unix was because Unix didn't allow them to have direct access
to storage devices, to disks in particular, which goes back to a decision made at the really
at the very beginning of Unix when they were trying to fit it into eight kilobytes of Colonel
memory or whatever it was back in the day on a PDP seven or whatever it was.
And there was a lot more control in Malthix and you know as the AT&T guys famously stripped
down to the absolute minimum system, they throughout a lot of stuff that all contemporary
operating systems could do and it's taken a long time for the Unix world to kind of catch up with
that which is I think is an interesting bit of history. So I'm going to go through four things
that all databases want from a from a disks subsystem. You can see this in Oracle and DB2
and SQL Server and MySQL, everything. Postgres is fairly late to coming to this game of trying to
do direct IO and the other three things that make up this sort of group of four things that we want
from disks. So the first thing is direct IO which means not using Colonel Buffers for data.
So on the left you can see Postgres reading a bunch of blocks from disk and pulling them all the
way up into just by calling the read system call or PREV or one of those kind of system calls.
Pulling that through the Colonel Buffer Pool and inter Postgres's own Buffer Pool.
It's fairly unusual for software for user space programs to have a Buffer Pool and that's kind
of where a lot of these special requirements come from. Almost all applications don't have these
requirements right. So we have a lot of opinions about when IO should be happening and the rate
of IO and all kinds of stuff like that that normally most applications just lead to the Colonel and
we don't really want to have a secondary Buffer Pool that we're kind of fighting with for
resources for RAM. We also don't really want to have the CPU involved in every read or write.
We don't really want to be copying stuff in and out of the Colonel PageCash or Buffer Pool as
you might call it. I think I'm using Linux terminology when I say PageCash maybe.
And so on the right, unfortunately I have to slide, sorry, scroll.
Using direct IO means that you basically just turn all that off and you hope, depending on the
file system and drivers and so on. You hope that all of your IO becomes DMA transfers directly
transferring data from storage devices into or out of user space accessible memory that's
mapped into the space in this case, the PushBestMuffable. But the interesting thing about
direct IO is that it's both an optimization and a pessimization. It's an optimization in the sense
that the CPU is not involved. It just builds a NVME or SCSI read or write command and sends it to the
device which means pretty much putting it into another queue somewhere and telling the device to
do things through the stack of device driver and so, a cam system and device driver and so on.
But unfortunately if you use synchronous calls to do direct IO, you then have to wait the longest
time possible. There's no other cache helping you. So every single read or write call is going to
take the full latency store. So direct IO gets a whole lot of CPU out of the way and gets a lot of
cycles out of the way. But you then have to stall. So that's bad. And we'll address that with
slide after this one. The second of the four things I wanted to talk about is vector IO and this
is a thing that or scatter gather IO. Not that many user space programs care about scatter gather IO
because normally if you want to read a large amount of data into memory you arrange to have a
buffer that's the right size and you just do your read or write. But databases because they have
again because they have buffer pools. They are really opinionated about where memory should go.
So they have a bunch of buffer replacement algorithm problems that exist also in the current
with the kernel's buffer pool. And the buffer is that they might find or might be able to allocate
to the data that you're trying to read in might not be contiguous. And yet we still want to generate
the minimum number of IO's. If you're using a cloud provider you might be paying for a certain number
of IO's or if you're using physical hardware. This is a certain number of IO's you can do.
It would be ridiculous if you had to do multiple IO's. Generate more IO's and take longer because
the buffers that you find to put the data into are not contiguous when you're reading a
large contiguous chunk of disk. So we want to better generate scatter gather IO commands at the
driver level and have them executed as a single DMA transfer straight into your non-contiguous
buffers in memory. And all the drivers can do that. That's part of the SCSI and NVMe protocol
and so on. It's just something we want to do from user space and not have it get lost somewhere
in the IO's stack and converted into a four loop that's actually doing multiple IO's.
So that's the second of four things that AWS is one from a disk storage stack. The third thing
is asynchronous IO and we'll talk in a moment about what that looks like but this is about separating
the submission of kind of starting IO's from the completion which means finding out the result.
It's an error. Possibly waiting for it to happen if it hasn't happened yet but if you align or
if your program is sufficiently clever and it does things ahead of time then hopefully you get
to consume the conditions later without going off CPU. So the goal of SXRSO is really to be able
to use direct IO effectively to hide the latency problem and solve that customization problem.
I think it's really interesting that so many other operating systems are even included
a meagodoss. That was one of my early learning experiences. Even meagodoss had async and as IO's
as a first class thing that you could use in an application so I remember games where it managed
to get a couple of floppy drives going at once without difficulty because it wasn't blocking
in the way that tradition. I have to speed this up because of the debacle with my Mac video
not working. So the fourth thing that I've grouped together is four things that are a little
bit special about database disco is that we want to make sure that all rights can be done concurrently
so we don't want to have if we're writing data out of our buffer pool to disk and it's consists
of many separate chunks. We want to control the amount of concurrency and we don't want anything
like a, you know, like an i-node level lock to cause those rights to be serialized which is
something that happens in a lot of operating systems when you're using direct IO that becomes
terrible because serializing a bunch of things is an incredible, you know, actual hardware
transfer waiting for completion will be very bad. Another example that so that's when
writing from the buffer pool the other kind of disk IO that databases all do is writing logs
and this it's a, I think it's a fairly complicated sort of thing but to get the maximum amount of
throughput and lowest latency that there's some kind of heuristic where you want to get, you want
to be able to start writing a chunk of transaction log and then more transactions happen and you
start writing before the previous lock had finished you start more and you want those to be
dumping how many as well. So that's an area where a lot of file systems have funny locking
and moidal out that to happen for various reasons. So that's the fourth of the four things that
I think are a little bit unusual about database disk IO. There are some more things that I think
which should have, I've done some experimentation in this area since times running short I'm actually
going to skip the set on the left there and say that being able to register buffers and just
the stuff on the right, being able to register buffers and register file descriptors so that they
don't have to be pinned or held each time you do a read from a file if you've opened a file and
and you're going to be doing a whole of reads from that file and each time you do that if you call
a reads system called inside there you'll see that the file descriptor has to be looked up and
and like reference count adjustments. There's some small improvements like that that can be done
and you can see that happening in Windows or Linux in very high performance programs where they're
really getting down to removing unnecessary CPU work. So I'd like to be able to find some
programming interfaces that would allow all of those tricks and these other tricks over here
that I haven't got time to talk about. So the reason I'm interested in all this is because in
Postgres 18 which just came out we've finally released something that we've been working on for
a few years which is a subsystem we're in the process of converting all of Postgres's disco.
Eventually we'll do network as well we have prototypes but I'm not talking about network here
it's pretty similar in many ways but I'm just going to focus on disks. So the idea is that
in the past I'm going all the way back to 1986 we did all our other by simply calling reading
write or p read and p write and in the last couple of years we've released in a series of steps
support for doing VectorDO and in 18 which just came out recently we've got this thing where you
can set this new setting IOMFID. On Linux there's an option IOU ring and that will use this new
kind of universal system call that they have in Linux which allows you to start all kinds of
operations and the interesting ones here are just reads and writes and that's done without
well in the best of cases it depends on the file system and all kinds of other things like
this compression involved this might not be true but in simple cases when you start an IOU it
doesn't run any kind of like a kernel thread to do there it doesn't use all the traditional code
files that would implement a synchronous read or write it just sends the you know converts
your instruct you know converts the logical blocks into physical blocks and does the you know
bureaucracy like that and then it pretty much just starts the I would pushing it down through the
driver and returns control and then later you can you can you know wait for the completion events
and all that happens without the assistance of any extra you know any scheduling overheads due to
internal you know worker pulls or anything like that there's nothing like that in the best of cases
which is what we want to what we want to happen we also have IOMFID equals worker and this is what's
used at the moment on free BST or any of the LBSTs or LIMOS and that's just a simple portable
system that we need as as a baseline and that has a pool of worker processes that do the IOs for you
so they finish up spending their time just consuming from a queue and sitting in reading write
calls during them for you so you can get on with doing other stuff. I've gotten experimental
POSIX AO patch which works only on free BST and I'll talk about why that is I tried to get it
working on whole bunch of other Linuxes you would think from the name that it would be portable but
POSIX AO is I think essentially a failure I've got a few theories for why it failed I mean
the basic sort of reason I think is that not that many applications or even programming languages
were ready to deal with ACIC when I say oh and not that many applications spaces kind of needed
that kind of stuff and the reason I grouped together those four things at the beginning of the
talk was to kind of explain that you know if you since common applications just don't really
have a need for a vector IO and don't really have a need for the direct IO which is a
pessimization unless you also have a whole bunch of other infrastructure to drive enough concurrency
it means that ACIC was also just wasn't that interesting from typical applications so
and you know there were a few big database companies and they in the 90s they figured out how to
use ACIC's POSIX AO on the big unit systems but if you look closely they didn't just use POSIX AO
they also secretly negotiated with all the commercial units vendors and had them add all kinds
of secret interfaces so they could make it work properly I think we need to find the right
set of interfaces and do them in a nice way that ideally would work the same way across the
BSTs and Lumos they have so much code and common that it would it would be a shame if they
can shut in compatible yeah I also experiment with some other operating systems that I'm
relevant to this talk so yeah work a mode is what you can see if you use POSIX AO on any of
these operating systems you're going to see these IO work processes running and you can
envision 18 you have to get to say how many there are in 19 we're going to try and make it so
that it automatically scales and yeah so that's where all the just reading rights are going to be
happening on behalf of the normal query execution processes which will hopefully be threads
as well but that's a separate topic the moment this is a process is so
sorry about this I managed to copy in place when I was making the PDF some of the
wrong slides okay I think my time was up so I'm just going to very quickly come to the
I'm going to skip a lot of slides and come straight to the kind of
question topics like I'm trying to find I'm trying to make contact with people who would
you might be able to help make decisions in this area so this kind of two levels to do really
good asynchronous IO and all these systems this kind of two levels that need some significant
hacking and at least decisions there's the user space level where I think the main choices are
either to actually implement IO uing or subset of IO uing as it's been defined on legs
either exactly the same for compatibility or something generally similar where a thin layer
could could work the same way or to look at the work that was done on KQ KQ supports
consuming completion events through KQ the funny thing about that is that of the five operating
systems that took three BST's KQ system KQ's great I love KQ but all of the other systems like
Max and Neppiosti I can be a ste they took KQ but they deleted the completion event support so
they can only really do readiness for sockets and so on all that works fine and that's very widely used
across a lot of software but it means that it looks like you could write something that would use
KV for completion events but it wouldn't be portable for those other systems I was a little bit disappointed by that
and then yeah so the choice really of like how we should respond to IO uing to me it seems like
either we implement it or we figure out how to make KQ able to do all the things you need to do to do
that stuff efficiently and I'll just finish up with the final thing which is so that's the surface
level question of how we should be done we can also do both of those things it might be that you
can do KQ make KQ a little bit better but also provide an interface that lets software be ported
and from Linux that uses IO uing sound faces I'm not sure about that but that's not really the hard part
I'm just going to super quickly the last thing is oh sorry please no it's just we have a
two minutes before the next yes sure so I'll just close up okay yeah okay yeah yeah yeah we are
running very very well okay so I'll just say goodbye then the final thing is to actually drive
asynchronous IO all the way through to the driver we need to change the VSA for VFS layer and that's
going to be kind of tricky the net BSD guys seem to have a project plan for that I would like to
figure out how we could make that work the same across the extremely similar code in previous
D and presumably open BSD as well if anyone knows anything about this project I'd be interested
in talking to them about it and that's it yeah