We're going to start again, just please get seated, or if you're chatting, maybe go outside.
Yeah, we're going to be talking about data access, monitoring, and access to where system operations now.
Can you hear me?
Okay.
Yeah, thank you for joining this talk.
So, my name is Song Jae.
You can just call me SJ, and currently working on mirror.
So, today I want to introduce a corner subsystem called Daemon, which is for data access monitoring and access of the system operations.
Let's talk about the data access monitoring first.
First of all, how we can define the data accesses.
We can define data accesses as events on space and time of memory that acting as fetching or writing data to some specific place on time.
And I hope we want to do data access monitoring in very precise and complete and light manner.
That is, we want to take time-generating edges as possible.
Hopefully, CPU cycles, device by number of CPUs.
And for each of the bits, we're even electron.
And for complete history starting from Big Bang, we're at least from booting of the system.
And we want it to lightweight enough to run on production systems.
But the reality is that it is very expensive and also you aren't going to really need it because the overhead is in order of memory size for time complexity and memory size.
But applied by the total monitoring time space overhead if you want to do the recording of all the events.
And maybe you wouldn't really care if a bit was accessed without five years ago.
And therefore, we introduce statement which is a tradeoff between accuracy and overhead.
It aims to be scalable in terms of a size of the system and provide best efforts of accuracy and being the lightweight enough to be used in real world systems.
And we start from the region-based phase handling for reducing the overhead of the space management.
We first define a region as an access motion unit which is a sub-area of the memory in terms of space and the time.
And it is a collection of average percent elements that have been similar access pattern or access frequency.
And instead by the definition, we don't need to check the access to each of the components of the region.
But we need to check access to one of the one random element of the region.
And therefore, for example, we can say if the page is accessed within last one second or not, regardless of how much of the cache line inside the page has been accessed or not.
So we can start from six granularities phase and time access region.
That is, this is sort of periodic six granularities monitoring like what we can do with either page tracking.
And in this game, the time overhead becomes mammalized, divided by space granularity.
And the space overhead becomes time overhead multiplied by monitoring time, but divided by time granularity.
That is, it can reduce it by the granularity.
But still ruled by the mammalized and total monitoring time and therefore, not that scale of it.
And therefore, we can just think about aggregating the bindings from the access sampling by n.
That is, we do n-graded access frequency, we define the sampling interval and regression interval and then check the access of access to each region in sampling for every sampling interval.
And then, aggregate the bindings for given user given time interval code obligation interval.
And then, please start for next obligation interval and so on.
In this way, we can let you just know more fine-grained access frequency with some lower overhead.
That is, the space overhead is now reduced to 1 divided by n.
Though the sphere is mammalized, multiplied by the total monitoring time.
And from this graph, we can show that there are some inefficiency.
That is, there are some other some three regions of similar hunties by the definition of the region.
Region means that all the entities inside the region had in similar access patterns.
And therefore, in this graph, the two regions of access frequency 1 and the two regions of access frequency 2, which are just inefficiency.
And therefore, we introduce a dynamic space granularity mechanism, which,
liquidably, merge the other three regions having similar access frequency and therefore, just wait for into one region.
And also, together with that, we randomly split each of the regions to two sub regions.
In this way, the number of regions becomes sent to the number of different access patterns of the system, or workloads.
And further we can let users to set the minimum and maximum number of regions.
The two regions could be the five regions could be two regions, right?
And therefore, we simply count how long the given hunties has kept for each of the regions using another counter-code age.
And then the snapshot contains history of not full length but useful length.
For example, by beginning each of the regions we have the age 1 and then if the access frequency has not changed, then the age counter is increased.
And from this point we can just discuss the first snapshot and we can relatively long history using only the final region, final snapshot.
In this way, Daemon provides the data access patterns snapshot like this. Let me give you a picture.
So we can start Daemon with, oh, okay.
Actually, the Daemon is running now and therefore we can just ask Daemon to show the access.
Pattern like this. So it shows on which others how long how big size of regions are there and how frequency access for how long times.
For example, in this case, the 409 megabyte region at the end of the other space is that access number for more than 20 hours because this is to come like the access.
And let's see the access pattern of gain.
Oops. It's already a bit.
And then it shows some more colorful outputs and also you can sort the regions by access temperature to show cold regions first and hot regions at the bottom.
So you can draw some kind of histogram that showing how much of the regions are in which specific access temperature or more intuitive way.
So that's what Daemon is providing. It provides the next number of regions times space overhead and both the time and space overhead are not ruled by the value size or more time which you just cannot control in many cases.
And how accurate and how heavyweight it is, it is up to your setting that is you can do at least control and under the control.
Daemon provides the best effort research and it is constantly being successfully used by some real world production products for years and also usually three to four percent single CPU usage is repeated so far Hawaiian skins.
The skins describes what memory operation options the users want to be applied to regions of specific access pattern for example find regions or that not access data for five hours and then reclaim those even if there is no memory pressure on the system.
We'll find a hot memory in a very far normal mode and then the hot memory in very far normal mode and migrate those to more closer but closer normal mode.
And then Daemon Daemon finds the regions of access pattern and then apply the memory operation action for each of the user given time interval.
And also there are some additional features for production level controls that is.
As I just mentioned Daemon can allow you to operate systems in access pattern.
Access pattern is not the full information that you need. You need some more information and some more ways to control it.
For that we have introduced feature called Daemon filters which defines the target memory of the skins in non access pattern based information.
For example, you can ask Daemon Daemon's to page output pages of specific mode node that also see I did with C group A and feedback like this.
This is important because you just might know something more than corner and also in some times Daemon's can be too aggressive will less aggressive than ideal and the users can set the resource portal for Daemon's using or feature for Daemon's quarter.
For example, you can ask Daemon to page output pages of 100 megabytes per second for using only 2% of CPU time and from this point we also introduced auto tuning feature of the quarter that is the quarter feature is important because it can control the aggressiveness of Daemon and the aggressiveness is what many people really need.
They want to page output page specific amount of port pages which is called this one first and in such a way quarter was but it is hard to know ideal aggressiveness.
The quarter auto tuning feature allows you to set only target system metrics and then Daemon is the auto tuning of the aggressiveness to achieve the target.
And we also found that Daemon has started as a way to operate systems but also we realize that Daemon is also useful for monitoring that is Daemon's can be used for efficient and fine grind data smothering for this we have introduced a special Daemon's action called Daemon's tab.
This is not a usual action for making real system operational change but just a special action that making no system change at all but just expose the skin information like which older regions were found as the Daemon's skin target based on the filters and photos.
In this way for example we can do page level properties based monitoring for example.
So let us begin and we have shown the access pattern in this way before right and.
By adding filter say resulting known on pages we can show how much of each of the regions are backed by non anonymous that is five act regions.
For example by just including Daemon that age had a file and then you can use Daemon functions and structures.
And also Daemon provides user space API based on cfs it is recommended for user space program development developers and also we provide the Daemon user space tour named Daemon which built on top of the Daemon user space API it is mostly commenting for you to use if you are human.
And finally there are some Daemon modules that written with Daemon kernel API for specific given purpose like practical information.
These modules provides more simplified interface and for those are also recommended if you if you want to use Daemon for the specific usage.
If you want to do first then you can also start from running Daemon test suit which is country being run on a very humble CI system for each of.
And also Daemon is committed to live on project of course because it is a part of links corner for any questions help touch reviews please use the community we have public value list we have 5 weekly virtual middle series which.
We had on google meet and also we have a project website collecting every resource and use and also the future old Daemon is there for always open and up to you because it is led by community so.
If before end of evolution over intelligent design and I also didn't know how Daemon will be like this when I started it and I have no idea about how it looked like a few years later so if you have anything you want please make change.
So in summary that's Daemon Daemon is a concept system for data assessment training and access over system operations and the future is up to you please participate.
That's all. Thank you. Any questions?
All right questions.
That looks great. My questions in the process and in progress there's you've got the pages and the messages that tell you the kind of memory ranges that each process has mapped and the output that you showed there does the kind of.
Here's the ground like compression boxing of your memory ranges over time and stuff is there anything that exists either written by you or somebody else that reconciles.
The access patterns to the memory that the pages that individual processes have mapped so that you can do reconciliation between process level.
Memory access and time series stuff.
So the question is if there is some use of space ABI like interface that shows so that is Daemon has initially developed for DRAM level memory access.
And in case of casual it might need some kind of cash level monitoring I believe was some other information and for cash level monitoring Daemon might be too big.
But even in the case in modern systems having relatively large amount of cash size that is it is not very rare to see about 100 megabytes size LLC cash right.
In the case even page rendering monitoring research could also show some pattern on the cash address space and for I believe that there could be some chance to use Daemon for scheduler.
And also currently it is using only access to be tracking however we have some plans to make it further.
It's tended to use some fine-grained information like how do a feature like AMD IBS instruction based sampling or a paid for each based information like the auto numer which can allow us to know from which CPU the access has made and based on that I believe that the scheduler could utilize some of the information hopefully.
Thanks.
Yes, I want to ask I think the swap system also keeps track of these recently used pages to what to swap out to what extent.
Is that completely separate from Daemon now and can you maybe merge those two sort of.
I'm currently it is completely separated that is we don't Daemon doesn't want to interfere the swap mechanism and therefore Daemon has developed to never interfere the swap.
Nevertheless, the swap system is using error mechanism to find which page to evict and Daemon is not touching the error list at all and therefore they are separate things and they don't interfere each other and also there is other page flaps that Daemon is using.
Besides what reclaimed logic is using, however one thing that we can further think about is using Daemon to predict claim called pages in this case the reclaimed logic is not directly affected but could get some benefit from the change and also we develop a Daemon's scheme that can find hot pages and then moving those to say.
Head of the error list and find called pages and move those to tell of the error list so that the later reclaimed logic can get some benefit finding better candidate of swap.
Yes, very cool.
All right, let me finish.
P-based on the access frequency and it has shown schwm benefit on the experiment and also there are THP shrunko which is split T and THP based on if it is assumed to be really used or not based on the content of the THP.
That is if say 90% of THP is filled by zero by it then it means that the user is not really writing the page and the version to be not used and therefore we can split those into based pages to reduce the internal fragmentation and memory overhead.
I believe that we can use Daemon together with this for finding nice threshold for splitting what that of the ratio of the gel-based pages in such case and also there could be some more other approaches I believe.
It's done if anything else.
All right, thank you.