All right. Sounds good. Okay. Good morning, everyone. On this lovely Sunday. My name is Sam
Cheng. I'm the global lead scientist for Impact and Evidence at the World Wildlife Fund.
I'm joined by Larry Kilroy, my colleague at Data Kind. And we're really excited to share
some of our reflections on developing a free, open, evidence synthesis tool for the past decade or so.
So what we're going to do today is just talk a little bit about where did this come from? What
was the challenge this was trying to respond to? Our experience is developing it, maintaining it,
Larry will run us through the technical development of this tool and then talk a bit about what's
coming next. Okay. All right. So we all know the world is changing pretty fast. And at one point,
a reporter in the Washington Post said, what if the solution to all of our world's greatest challenges
is already been solved? It's just hiding in a PDF that nobody is ever going to read. And they
published this article because there was a report from the World Bank over ten years ago
that out of the tens of thousands of reports, they published every year about their projects,
about policy analyses, only a third had ever been downloaded, which does not mean they've been read.
And so if you take that alongside the fact that research output is rising exponentially. We're
getting new papers published every single year. It's on tracked double every nine to ten years.
So on one hand, right, this is really, really exciting. There's a lot of information and
evidence that's out there that we can use as practitioners and policy makers to inform our work.
But we need that information in a timely way, in a digestible way, so that's usable,
so we can actually feed it into the decision-making processes at the right time.
And this is where approaches called evidence that this is coming. So this is a really broad
set of approaches that exist out there. They help us capture review and synthesize existing knowledge
for specific topics and questions. And so this really came from the medical field. So anytime
you go to the doctor, they might make a recommendation. That's informed by medical guidelines.
Those medical guidelines are updated based on publish literature on a regular basis.
And what's really valuable about these approaches is that they're transparent.
They aim to minimize any potential bias that might be in those days of the fact,
the types of insights you get out of them. And they're also reproducible, meaning that you can
repeat that analysis at any time so you can update the evidence base to keep up with all that
stuff that's getting published. So at WF, we are a science-based organization. We are trying
to develop solutions to answer key conservation challenges so that we can halt the degradation
of the environment, conserve biodiversity, address all those drivers that impact nature,
including climate change and human development, and ensure the well-being of people.
But to do that, we need timely access to reliable and relevant information at the time
skills that we are trying to make decisions that are tailored for the places where we're working.
So in response to this, I feel like I'm aging myself here. Like 16 years ago or more,
my colleagues and I across a number of different conservation organizations and development
organizations said, okay, let's take advantage of the decades of research on conservation.
So we can understand what do we know about the impacts of nature conservation on people?
We thought this was going to be a weekend project. So there's that.
And we ended up looking at 35,000 published citations. Taking all that,
winning it down to about 3,000 relevant articles and getting to about 1,000 included in that map.
And there were a lot of really exciting insights there, which helped us understand where we had
known and known unknowns and gaps we needed to address. But it took us three years. So by the time
we did that, our knowledge was three years out of date. And to do it again, it would take us another
three years. And so it very quickly became a very daunting task of figuring out how we could
actually use this on a regular basis to inform conservation work.
So that leads me to the second part of the talk. To talk about the challenges of synthesis,
to really make them scalable for decision making on the whole. The first piece is obviously the
efficiency. We are taking a lot of time. We have to sort through a lot of stuff to get to that
proverbial evidence needle in the information haystack. And so to do this process manually
is quite literally impossible. Which leads me to this point that not only are they inefficient,
they're also sometimes insufficient to look for the right information. We look to cross the environmental
sector. We found that for similar types of studies, authors are looking at around 14,000 citations
to get to really only 2.5% of those. They were actually relevant to what they're looking for.
So obviously there are a lot of reasons why you might not include things. But ultimately what the
verdict is here is people are looking through a lot of potentially irrelevant stuff to get to what they
need. Which leads to the second challenge. It's really costly. So all that time it takes to search
through all those citations costs money. This is a fairly old study, but I think still quite
relevant, not accounting for inflation. But on average 30,000 to 300,000 US dollars to do one
synthesis, not updated. And it can take anywhere from six months to a few years. And so that's
not really sufficient if we want to generate evidence at the timeline that we actually need to
make these decisions that. You can imagine somebody in a government agency is like we've got
a past as policy, we've got a format in the next couple of months. And we're like ask us in three
years and we'll help you. And so any sort of approach we take to making this process faster,
we also need to take account the third challenge. Which is the value of these evidence and
this is approaches is that they're trustworthy, they're rigorous. And so whatever the outputs
of any sped-up process is, we need to make sure that they still maintain this component,
because the value of these approaches is that you know the evidence hasn't been cherrypicked
to match a particular decision or finding or position. So it's within this context that
calendar was founded and launched in 2017. So what is calendar? So with the science for nature
and people partnership that was led at the time by conservation and international, we decided that
evidence synthesis was something important that we wanted to pursue and support across the conservation
sector. But we needed to find a way to do it faster, more efficiently and dynamically. And so
we partnered with data kind to develop an open access, open source computer assisted software
platform for evidence and physicists called Column, or because you're sifting stuff through
to find what you need. And so the way the calendar worked was really to have an easy to use
front-end platform where users could input what they were looking at to help them sort through
faster and intelligently labeled that information. So when we developed Column,
we had some key features in mind. The first is that we had to maintain a human in the loop.
It couldn't be fully automated. Again, the value of these approaches is that they're transparent
and we wanted to maintain that trust. So one of the features that's built into Column
is an active learning approach where users are maintained through the whole process,
maintaining that oversight. And the second is we wanted to make sure that it was open.
So at the time that we developed Column there over 10 years ago,
there were lots of different approaches trying to apply machine learning to evidence synthesis
in different softwares. But we didn't know what people were doing. They were opaque,
no one was really talking to each other. And so every time someone tried to start something new,
they were basically reinventing the wheel. And this took a lot of time and resources for our
development impact and dive and figure out what they were going to do. But then the second part
of it was, was seeing that there's a lot of opportunity to apply these various different AI approaches
to synthesis, but we needed to foster some kind of community of practice to learn from each other.
And so that was the rationale behind pursuing an open source approach.
And the third is Column's open access. It's free to use. And this is really,
I think predicated on the challenge that if we want to be able to use evidence, we should be
able to use evidence everywhere. Everyone should be able to use it. Some of the biggest challenges in
the world, particularly for conservation, are in some of the countries with a few of the resources.
And so researchers and low middle income countries don't always have access to really expensive
research platforms and tools. Moreover, if we want to update them, they again cost money. And so this
is something that was intended to help accelerate the field forward. Okay, so in the past 10 years,
we've seen the user community for calendar really grow organically. We have over 6,000 active
users now, nearly 4,000 paths and ongoing reviews that are being conducted. And it's really been used
across a lot of different fields. We've built this for conservation, but turns out it was applicable
and answered a need across health, you know, education and social development, environmental management,
you name it. And so what some of those users have used Column for are folks like the ICO
Alliance who's been using a to extract information about certification impacts, right? Like four
certification, product certification, the American College Physicians has been looking at the
machine learning algorithms, but they're behind Column to help them improve their synthesis approaches
and medicine. And then for us at WF, we are working to harness some of the work in Column
or to help us figure out how we maintain living evidence and conservation.
But the other thing that's been really exciting is that this community of users has
helped grow some of the features, right? We built Column as sort of an initial tool.
People then came and said, this is great. We want to build some add-ons. We want to build tools
that can complement enhanced functionality, leveraging things like R so that they could create
trackers and create things that would pipe the outputs to other tools. They've been helping
track the research performance, conducting studies, comparing calendars performance to other
tools that are out there and providing feedback on how we could continue to develop it to respond to
changes. But this also raises some of the challenges. In 2016, Column was built by a team of volunteers,
working on their nights and weekends. And then over the past nine years, it's continued to be
maintained by these volunteers with sporadic support from various institutions, really following
me around to multiple institutions over time. But we've been lucky that we've had a really
supported and committed user community that has allowed us to continue to see the value and
maintaining this over time. But it's on that note that I'm going to hand this over
Celery to talk about the technical development and what's next for Paulander?
Okay, hopefully that helps. Excellent. All right, so the technical development of
Column. Data kind is an organization or a nonprofit organization and we're trying to close the gap
between the access that non-social impact organizations have in the user every day in social
impact organizations use every day. So this was an early application of machine learning and a lot
of the base machine learning we're using was key in the development of what we now think of
as Gen AI in large language models. And so Column was built as a two system approach.
One, looking at inputs of search results from existing systems and the other uploading articles
for full text assessment, and we'll dig into each of these. So on the first, we're using a
technique called word-to-vector developed by Google in the early around 2013 by Google researchers.
And it's a technique that transforms text into numerical vectors. We have a little example here
where it takes the I like NLP and I love cats and based on the association of the words starts to
create a scoring system. And so the way Column uses this is we use it to identify
co-occurrences of words around user-friendly key terms and we actually read the abstracts and
provide relevance to that. And this is what the system one diagram looks like.
The second technique we're using is referred to as glove, which is global vectors. So we're capturing
global vectors. Most of this work is in the English language. So capturing in the English language
and we are generating word vectors based on occurrence and relevance statistics. And then it
creates a probability of giving words what they actually mean based on how it's used in the
presence of other words in the system. So this is a slightly different flow. So we would read in
actually like a PDF and then apply this technique. So we aren't pulling any other information from
another abstract system. So all of this, and I'm going to jump, I hate to do this to everyone.
But thank you for your inspiration. We're going to do some live stuff.
We, that whole back end goes out to arrest API. So right now we have built a user interface.
It's based in English. But our hope is that with this published API, others can build their own
applications to it. So basically all the documentation is here. For example, it's easy to be
for you commit to building, to go ahead and we would provide any organization and tokens and authentication.
You can just try it out in this interface before you commit to writing any code.
So that's the basis of that, what we call the back end. And then we have built a front end.
And this is meant to make it easy to use. Here we go. Oh, there we go. It's working.
Loving we're living on time. I'll just show a couple of the key features.
Action, I'm going to back out. Well, I'll do this first. And then we're kind of going to reverse.
So here, you know, there's some basic tools. You could have quite a bit of these uploaded
citations, especially coming from a system. We might be hundreds at a time. And so this is meant
for teams to use. So teams of researchers can go through and use us. And you can sort by relevance
or you can sort by recently added. If we go ahead and pick one, then you have the information
you need to go ahead and start cataloging. You know, if you wanted to enter a tag,
I'm just going to say, I'm going to say water. Keep it interesting. This is the test database,
don't worry. And then that allows you to eventually filter by all the different tags that
have to do with water, etc. And then, of course, you can go ahead and include it or exclude it.
So similar to the system we just saw. And then once is included or excluded, it'll be, you know,
in the completed phase. If it's in conflict with another reviewers, it shows up in that conflict,
and then they can be resolved going forward. Again, knowing we are limited on time,
let's just go quickly. We want to take a quick look at, we want to make it easy for
importing information into the system. So, you know, we just have a basic import here,
get some data information, make sure you can identify the data source, etc. And then it's
pretty much drag and drop. So you can drag collections in here, things like that, keep that very simple.
And with that, I'm going to skip back because we're wrapping up here to, and I'll go back to full screen.
That was the human in the loop part. So that is how we maintain that human in the loop.
So what's next? So as Sam mentioned, it's been maintained by a very small group.
We're hoping to expand that. As we get more users, particularly more university users,
we've had requests, can we contribute to the code? So we wanted to make some changes. The early
versions were written in Scala, which is a JVM language, so not as well used these days.
By researchers, especially not used it all. So we've refactor that entire back end to Python.
Something that can be used. We've also, as you saw, that's the brand new API. We've refactor the
API to a fast API, which used to be called Swagger. And we've rebuilt the front end in ReactJS,
which is a popular, one of the most popular JavaScript languages. So it should be easy to
contribute. The big thing, anytime you're kidding, and we've also had a build out our test suite.
It didn't have a lot of coverage. So that way, it's easier for our team to test any incoming
pull requests and merge. Up on GitHub, MIT license. So people can take this and improve it.
The next thing we've done is we have through email primarily, which Sam has been the full customer
support. We have brought in all of those that we started creating a GitHub issues. And we're
hoping that, especially for people who are early in open source and want to pick, get their
public commits up. This is a great place to start, find an issue, solve the issue. And then finally,
we're establishing maintainer policies. Hopefully not the same for people who will be doing it
all the time. So how people can also help maintain and approve that coming in. And the last step
is very new, which a lot of the ML we're using has been used in creating LLMs. But now we want to
explore how LLMs can also supplement this work. And that'll be a big focus in the next year or two.
With that, thank you, and questions. And I'll make this mobile and so depending on who I say answer.
We'll start left to right.
So interesting enough, a lot of the feedback we got. Oh, the question was, thank you. That's
why I just put my glasses on. The question was, are we considering redesigning the user interface
based on feedback that we've gotten from caller one to caller two point two point out? The answer
is, there are a number of changes we've made because caller one wasn't up to date with all the various
standards like GDPR and others. So when we hacked those in, they became pretty ugly. So it's a
much better looking interface. But for the major features, we've actually got a lot of feedback
that the simplicity of the user interface is one of the, one of the selling points of this version
of caller. So we don't have any major changes planned, except for improving around like the
drag and drop technology, things like that, and making sure that that's obtained and obtainable.
We do hope that there will be contributions of new features though, and that if it requires
some adjustment, we would do.
I'm going to hand us the same, but the question was, where do the research articles come from?
Are we limited to open access?
That is a great question. So if you're a caller user, you have to use your own access licenses,
so whether you're at a university or wherever else, you are the ones searching for those publications
and uploading them into callnors, so we won't search the databases for you.
The question is, I mentioned callnors two is coming soon. How soon? Hopefully within the next six weeks,
we're in final testing right now, and then I saw a hand over here.
Yes, I some of who does conservation research, I've done it for many years, thank you for
what you do. My question is, I saw the words include an exponent on the demo that just now,
so it's what any particular paper would call and edit tell me, you know, callnors, why
it suggests that including the word exclusion.
The question is, is there is an include an exclude function during that review process,
and would we have the reasoning that someone else could access why it was included or excluded?
That is a great question, so because all decisions in callnors have to make by a person,
when you say include, you can, that you know, it meets all the criteria, your team should have met
and decided these are all the things it must fulfill. In the exclude, when you click on it,
it gives you a whole set of options. You can define all those options yourself, right?
So say you're interested in studying the impact of restoring mangroves on storm surge, right?
You get a lot of storms, you can reduce flooding, and you might say something like, oh,
this paper is about mangrove restoration, but it's not looking at the impact of storm surge
at reduction, so you'd say no, like irrelevant outcome or whatever, so you can really come up with
any reasons you want, you can add whatever tags you want, and that's to keep that whole process
transparent, but it's not going to tell you you should include this or you should not include this.
It's just going to say this seems more relevant to what you've been saying is relevant.
And I'll just add that's an area we're thinking about where LLMs could be helpful. It could,
you know, synthesize what all different teams have said about that, and maybe then we can maybe
have some sort of suggestion feature. Oh, yes.
Do you say open Alex? Well, one, I have not heard of open Alex, so I have not considered it.
So the question was, have we considered using open Alex as a source for reviews?
So I think that's, I think that's a great suggestion to the evidence and this is community, right?
So the choice of where you look for papers and where you're looking for a mainstream is really important.
You look in, you know, a really small database, you're going to have a small set of things to come back.
But I think that's something to go back to, okay, where are the open source resources?
We can be relying on as an evidence and this is community that are actually going to allow us,
more visibility to all the things that are out there. But all under the time doesn't pull from it.
Thank you. Thank you. So next up, we are going to stream a video from a speaker,
speaker, that couldn't make it because of visa consideration.