So we have another new topic today within biology that we're covering.
So we're going from pictures and recognising decisions between huskies and walls to behavioural ecology.
And so please let me put your hands together for, yep, and shall I go?
Yep, for nickel, simple arts, we'll be talking about movement, which is here this is a call for community.
So do you have a look and get involved?
Thank you.
Can you hear me okay?
Yep.
Okay.
Thanks for the introduction.
I'm very excited to be here at the first, as I say, no, the first pair of musics and computational biology dev room.
And it's a bit of a leap, because we're going from genomes and proteins, which were most of the talks to animals.
So my talk is going to be a bit different, but I hope it will be enjoyable for most of you.
So I will talk about Python Toolbook, we are developing, which is called movement, which is for analyzing animal motion tracking data.
So first of all, a bit of information about myself.
I'm a neuroscientist and research software engineer, based at the Sensbury Welcome Center in UCL.
That's the building you saw at the title slide. I work in a team of research software engineers.
We call ourselves a neuroinformatics unit, so just swap the bio for neuro.
I'm also a fellow of the software system ability institute, which was mentioned in an early talk.
And I'm the lead developer of movement, which is the most relevant affiliation I guess for today's talk.
So I'm going to give you a lot of background, because I expect that many of you are new to this field.
Before I jump into explaining the package, so the problem we are trying to solve is to essentially quantify behavior.
And this is tricky because behavior is a very nebulous concept to define.
But some definitions are useful, and I like this one from 1951, which defines it as the total movements made by the induct animal.
And I like the world movement, because movement we can measure precisely.
And how can we measure movement?
First of all, we can use video cameras to fill the movement itself.
That will be the topic of the talk today.
We can also use inertial measurement units, which are sensors you can attach to the body, the teva accelerometers, and stuff like that.
You can also use GPS-based biologing for tracking movement on large time scales and large special scales.
But today we are only talking about video cameras.
The problem with video cameras is that you somehow need to transform the image, which is the video frame, into motion, or the series of frames into motion.
The traditional way of doing that, that maybe you are familiar with this motion capture.
So in the entertainment industry in cinemas in animation, you would wear suits.
They will put reflective markers on you.
You will go into a special design room, which has cameras, I think 20 or 30 cameras all around, that film you from every possible angle.
This camera is located reflective markers. You do a 30 reconstruction, and you get a 3D time series of motion.
Amazing. High quality. The problem is this incredibly expensive.
And it's a bit invasive because you have to wear the suit and the markers.
If you're trying to do it in animals, that's not so easy and very, very costly.
So animal researchers have been resorting to more local ways of dealing with the problem.
You can see one of them in this paper.
You can attach color markers to an animal or different body parts of an animal, like an ant, in this case.
And then you can use traditional image filtering methods to identify this colors during your analysis.
But this has changed because in the last 10 years or so we have computer vision, deep learning based computer vision.
And the new kind of game of the name of the game is markerless motion tracking.
The way this works is that like everything else nowadays, you train a neural network.
To identify body parts. So basically you label some video frames by hand.
You click on the nose on the shoulder or whatever you care about. You label hundreds of this frames usually.
And you teach a neural network to recognize the same body parts in unseen frames.
If you do this for the entire video and you train the model well, you can give it a video and it gives you this representation of the body parts overlaid on top of the video.
There are many user friendly tools that to do that workflow.
Probably the best known one is deep log cuts.
But there are others like sleep and lighting posts and probably 10 others that do a very similar job.
So this is actually quite cool because it has transformed the study of animal behavior because it has changed the scale.
Before you have to click through every frame, now you can do it in a few days.
That's why essentially animal behavior scientists are now falling in love with computer vision.
And that has brought many different fields that were different together.
Because now neuroscientists by mechanics people, zoology, conservation psychologists, vets, animal welfare people, livestock farming people.
They're all interested in using these tools or they're already using them, which means that there is a convergence in the toolbox.
So basically all of these fields are now starting to use markerless tracking.
The problem is what happens afterwards?
Because this advent of computer vision has kind of moved the bottleneck one step down the line.
Because now we can produce this motion tracks very fast, but we are not actually interested in the motion tracks.
We're actually interested in the behavior of the animal.
So we want to essentially extract meaning or measure some meaningful methods of behavior based on tracking data.
Some simple examples would be maybe you want to measure the speed of the animal or the orientation of the animal in space,
which areas of the environment it has visited, how fight has traveled, things like that, or more complex things.
Currently this is done in ad hoc scripts in every lab.
So every piece is hidden or postdoc, writes a script in Python, MATLAB or R to compute speeds.
There are probably 200 versions of Python functions for computing speed in only my institute.
And that's not really great for the field.
So there is a lack of standardized data formats and tools because all the computer vision toolbox,
I mentioned they speed out data in different formats.
We have lots of fragile in-house scripts that are not maintained beyond the product's conclusion.
And we have piles of analyze data because we're producing motion tracks faster than we can analyze them.
So this is precisely the problem that we set out to solve and we are solving it with movements.
It's a Python toolbox, it's a Python package.
You can find it on PIPI and on Call the Forge.
So that's kind of the visual summary of it.
What's it does?
It loads data.
It ingest data from a variety of animal tracking frameworks, including the most popular ones.
And we are continuously adding support for more formats.
Once the data is in Python, we have a unified representation of the data,
as a multi-dimensional array using the Python library X array.
And this means that it doesn't matter which animal species we are studying, how many animals we are studying,
or how you track them.
Once the data is in movement, it looks the same.
This allows us then to implement a bunch of general purpose methods for filtering, visualizing, and analyzing the data.
And we do the implementations in a way that's fully tested validated and documented.
So what are the source of things you can currently do with our API.
You can check out also the examples on our website.
You can do that cleaning, which is very important because as we heard in the previous talk, models make mistakes all the time.
And you want to use some characteristics to identify these mistakes and correct them post hoc.
You can visualize the data, you have some plotting utilities for plotting the trajectory of the animal,
or which areas of its environment is visited as a heat map.
You can precisely quantify how much time the animal spent in each area of the environment.
You can measure the orientation angle of any body part you are interested in, let's say, here at the head.
You can also use it in specialized applications.
You can do ppulometry, you can track points on the pupil and then measure, let's say, the people of Lovs,
they as I move from left to right, and many, many more.
We are also interested in kind of lowering the barrier of entry for users,
so we are developing a graphical user interface as a plugin for the Python image you are in a parry.
What you see here is, I've load the video of two mice in octagonal arena inside a parry.
I can navigate and scroll through the video like you would do in a normal video player.
But now importantly, you can use movement to essentially overlay the tracking data on top of the video.
And you can do this from a variety of different software sources.
You can set the file path and then the data appears in the parry as two layers.
First, you get the points, which are the predicted key points of body parts.
And you can display their labels to inspect them.
But apart from points, we also have this other layer on top, which is called tracks.
And you will see what that is in a moment.
As I play the video, you see essentially this tail extending behind the animal, which is the position of the animal in the previous frames.
And you can very easily play with the parameters of that.
So you can make the tail longer or shorter to view more or less of that history.
That's quite useful for identifying tracking errors.
And you can also go the other way, so you can extend the head and see the future.
So if you do it all the way, you see the entire trajectory of the animal in this experiment.
How are we doing all of this?
So we are following a completely community power to commit the driven development.
We have a core of engineers, I'm one of them.
But until we have 34 contributors on GitHub, from multiple research slabs around the world,
we have over 300 much pull requests, so far, 80,000 downloads on Pipi, I'm going to forge.
And we already have several external packages depending on movement using it as a dependency essentially.
So the bigger picture here is that we want movement to become a core part of the analysis workflow for analyzing animal behavior.
That takes you from motion tracking data to medical metrics of animal behavior.
And we see movement as a key part of the scientific Python ecosystem.
I like to say, like a psychic image for animal motion data essentially.
But if you don't like Python, you prefer R and I know there are many of you in this room.
There is a sister project in R developed by a collaborator of ours, Mikael.
He calls it any movement.
And essentially the same workflow, and we work together to standardize on other formats naming.
So we kind of keep the development step by step and you get the same similar experience in Python and in R.
That's all I want to say today.
I would like to thank the other two core developers, Shankuan Lo and Sophia Miniano, who are not here today,
and our supervisor Adam Tyson.
Our host is in Toots and a collaborators, and of course the many users and contributors who have made movement.
And I will also mention if you're interested in that project, check out the links.
And also we are running a workshop in London next summer for people who want to learn how to analyze videos of animals with techniques like the ones I mentioned today, including with movement.
The registration is open for another two weeks, so you have time to tell people to register for that.
Thanks a lot for your attention.
And a question.
I also have some experiences that I've already set up here.
No questions?
I have one.
Do you package it about movement?
Do you want to be changed?
Do you want to be managed to create a bigger gate?
Or is it planned?
Yes, I will do.
So the question was that by the package is about movement, but what eventually you want to get at this behavior?
And there is still a long way from let's say speed to actual behavior.
What I didn't mention is the thing you sit here on the bottom, which is so called behavior segmentation, which means you take the continuous motion facts that
and you segment them into identifiable behaviors that you a human would describe.
This is not in scope of the project, but we are maybe working on a different project that covers this scope, because we want to keep this package quite lightweight and single purpose.
Thank you.
So we actually have a few more minutes still before.
So I might actually ask a very quick question, Nico.
It is trying to run off.
So low level data formats between R and Python using x-ray are just having x-ray implementations.
Yeah, we are actually discussing this with Miguel, who is the guy who developed the R package a few weeks ago.
There are solutions.
So one of them is x-rays native format is net CDF, which is actually just an h5 with a specification.
And low-d h5 in R, the other solution we are talking about is also transforming the data during x-ray in a tabular format and say we can pass per k.
So that would also be an option and we are working on that.
So in the end we don't care about having one single format as long as we agree on the spec, we can save any kind of file we want.
Okay, so thank Nico once again.
Thank you.