Hi, I'm Caroline, I'll introduce myself in a second, this is Antonia, and this is our talk
design in five minutes, okay, 20 minutes.
We're going to be covering ecosystem mapping and user research, or really say to be here.
We did design this talk for folks that are new to design, but if you have a long experience
with design, I think we have some really interesting case studies for y'all, so yeah.
I'm Caroline Senders, I'm the founder and principal designer of
communication, research and design, and I'm Antonia Valencia, the design researcher in
communication, research and design.
So today we're here to talk about open source projects, and many times when we think of open
source projects, this can be or seem a little bit like a multiplayer game, so there's
many contributors, everyone is participating, and it's sometimes beautiful, sometimes very chaotic.
You fix one bug, another one appears, somebody does a request at 3 a.m., but it's kind of the magic, right?
But for your project to really thrive in this ecosystem, you really need to understand
the broader picture, so where does it actually fit?
So again, an open source project really exists in a vacuum.
Everything is connected, you're building tools, libraries, platforms, you're contributing
to different web or other projects, technologies, and collaborating with people.
So design really has this capability of building bridges, so knowing where your project fits,
within this nice ecosystem, it's not something, it's not like a nice to have, but it's something
essential. Understanding where it fits, how you can collaborate, and the perspective of design
is very important for this building of the ecosystem mapping. So it's going to become let's hope
one of a key element in your design process. So we like to think of ecosystem mapping as a
calling of an API between your project and the environment that it actually exists inside.
So what are the dependencies? How does it connect to who are the users, the contributors, the
monitors, this is something key. So how do humans actually interact within this whole system?
So building this map of connections and identifying the relationships will actually sit
to a year project within this ecosystem that surrounds it, and it's also going to help you find potential
collaborators, competitors, even, and hopefully understand the interactions of use of your
users and of everyone around it. So now let's dive into the things that you can do right now,
that is going to be mapped out your ecosystem, and also try to debug this human experience and
interactions. And then now we're thinking about what actually does it take to do ecosystem
mapping. So this is where I think it's important to define this opening objectives of your project.
A lot of you probably do that, but it's always good to write it out explicitly for the entire team
you're working with. I think the next step is important to then figure out who are the key stakeholders
and communities that are either going to be involved in the project or will utilize it.
Then I think it's important to analyze the interaction, and by that is how people engage with it,
but also what are potential interactions with similar projects with dissimilar projects?
Where do you sort of sit in a space of let's say the entire community? Where are their places to
leverage collaborations, and where are there also potentials to sort of say like actually even though
this seems really similar to us, we're actually doing this very very different kind of thing,
even if it's on let's say like it's a similar programming language or similar again type of project,
and then lastly documenting findings and insights for your team. So sort of taking all this together
and presenting it back to think about how do we position ourselves not only inside of an ecosystem,
but also how do we talk about with our community where we think the project also fits?
And we're going to get a little more into this. So these next steps are going to focus on
it's almost like a deeper dive into steps two and three. Mapping just isn't about
coder dependencies, it's also very much about people. And thinking about people is this experience of
thinking about these invisible bugs. Why aren't new contributors contributing or sticking around?
What are the barriers they're facing to engaging with a project? How do you balance sort of
onboarding let's say new community members who have a variety of different backgrounds or skills?
Does your project have space for less technical and more technical folks? How do you want to
engage with them? How would you like them to engage with the project? What are barriers for people
getting understanding the project? Is it just language or is it how to actually find let's say
materials about the project? I think all these things are really really important and this is also
where we're going to get into user research to understand your community more or understand
who's actually using the project. So user research tips and tricks. So what is user research?
Something we could call qualitative research, are like to think of it as talking to people or
engaging with your community. I'm involved things like interviews, focus groups, surveys,
and some cases you might do something called diary studies where folks will be sort of writing
down how they're engaging with something on a day-to-day basis, but also uses a variety of other
tools. And so here's what I kind of or what we sort of think is the the bearist of bones
of user research. And it also I think helps to define like why you would do it. So it helps us
identify a problem or focus area to outline what it is you're doing. It helps us understand why
I like to often say is how do you know how do you know that this is a problem. An answer could be what
we saw people complaining on GitHub or to our volunteers or we just saw things online across a variety
of spaces. And so then we spoke to a few people that complained or had thoughts. This is a great
way to sort of validate how do you know. But it's also the one I think one the most important
things. There's a problem how do you know it's a problem where did it come from? And then the next
step is responding to that problem to create a goal. And what I mean by goal is like what what
are the next steps? What would you like to solve? How do you start to distill that into let's say
an area to focus like a question or just really sort of narrowing narrowing that down. So a goal can
be like we know that users keep repeating the same mistake. Maybe it's maybe when they're engaging
with certain API using too large of a dataset. Why are they doing that? And you know then it could
be and we know this because we got X complaints. So the goal might be we need to learn or understand
better how this mistake keeps sort of refracting inside the community over and over again. Our goal
is to understand more and try to come up with a solution. Then the last thing is a plan.
Or not the last things. Sorry, but then you can create a plan towards actually engaging with
users and interviewing them. That could be we decided to run a short survey or we decided to speak
to like 20 users, some who are much older users who are maybe higher impact contributors
and newer users. What are their experiences and how do they differ? How does someone have a lot
more knowledge about a project than than others, right? Then it's from that plan synthesizing results.
What did you find? What are the patterns? What are the patterns tell us? And so then
instead of taking all of these things for next steps to think about what a solution is. So validate
findings, right? Say this is our thought of next steps of our solution. What does the community think?
Ask for feedback. And then so on and so forth.
All right, so now that we got a first look at user research, we go back into ecosystem mapping.
So this really comes into play when you start understanding who is around your project,
who are the users that are going to be interacting with it. So here are the tips and tricks
for ecosystem mapping. To make this ecosystem thinking part of your process, you first need to start
mapping early. So don't wait until your project has so much complexity that is going to be hard
to jot it down on a piece of paper or a map or whatever means you end up using. Then you can keep
it simple. So you can even do a hand drawn map to try to put down these interactions.
Then share and update these maps. It's a live document. It's never going to be a static piece.
And finally involve your community. So mapping is always better done with different perspectives,
different opinions, and start doing all of this collaboratively. So why ecosystem mapping? So
this tool really helps you understand what else is out there, what is similar, what is
this similar, what makes your project stand out between the rest. It also helps you understand
challenges and that other projects have already undergone and how they really took a particular
view or a focus. It also helps you visualize, so visualize the gaps in this ecosystem and find
the opportunities where your project or your community could try to help fill and it allows you to
understand and explain the why and why this project in particular, whether the characteristics
that make it different from the rest. And finally it really allows you to engage in a companionship
in working together to build this tool and building your project in a more wide view.
So to further understand this tool, we're going to present two case studies. The first one of them
is the Red Decoratia Hital, which is a project done with school teachers in Puerto Williams,
Chile, together with the Favla Baustral, which is the Favla in this island in Chile.
And this project was situated in a very particular context, first of all, it was in a remote
island, at the end of Chile, in the southernmost Favla been the world. So it has very particular
characteristics. This island is very small but has like very different stakeholders inside of the
island. And we use this ecosystem mapping tool to try to comprehend the context around technological
education. So what are the gaps and opportunities to create this more profound implementation
of these skills? So the subject of technological education is immensely large and it is a domain
that we wanted to get our hands in, but this tool really helped us that the ecosystem tool really
helped us to guide us in which direction to actually take. So the first insight that we found
is that we were going to work directly with the educators. So by doing this, it would enhance
their understanding of the technological language, the tools, and thus they would take these skills
to then design the curriculum to take to the classroom. This also allowed us to onboard different
educators from the island that in this particular case, this only one school and one pre school
establishment. So getting together with a couple of teachers would make the rest of them start
getting involved and onboarding themselves to the project, inviting others to collaborate. So
also creating a community between them. And after this, we also thought that we wanted to identify
different hubs and technology clubs around Chile, being a very long country and them being
more in isolation. We started collaborating with different of these actors around the country
and also introducing this technological hub that we are trying to create in this broader network.
So at the end of the day, a ecosystem mapping really allowed us to define the scope,
identify the users, the collaborators, and also to visualize these gaps that we had found
to create the outputs and actual implementation of the project. Great. Thank you.
I'm going to talk about another project that I worked on. So I used to be a design researcher at
the Wikimedia Foundation on the Anti-Arassment Team, which is a sub unit of the trust and safety
team. So one of the things I worked on there was called the Interaction Timeline. So this project
I'm going to talk about covers both ecosystem mapping and user research, because Wikimedia itself
isn't incredibly large ecosystem. We're talking hundreds of different language wikis,
Wikimedia Commons, things called village pump, etc. These all serve exist inside of this one, one space.
And one of the things that we actually utilize inside of the Wikimedia community, something
at the time we called the Wikimedia Community Wishlist. I mean, have y'all heard of this.
So the community wishlist was a way that across the year different community members could list
things that they would like to see. The team I worked on there specifically focused on. I was
on a team that was called like community and building. So that would be like if you saw a bug on
something you really wanted to fix, if there was a new feature you'd like to see, or if there's
something that we should focus on. And then we would look at how people had sort of voted or ranked
all the things that ended up on the wishlist. We would go internally to our team and sort of talk
about, okay, well, how difficult will it be to implement this one thing if it's really easy,
even if it's lower on the wishlist, we are going to sort of like maybe prioritize building that
really early on so take a short amount of time. Other things might be very, very important but
might take many years. And so what we would do is we would then present back to the community,
the wishlist and say, here's the order of operations we're going to go in and then here's our
rationale. So this one thing you really want, we're going to take it back to like the board of Wikimedia
and talk about how we actually need a fundraise for this thing, but it's maybe not something we
can build this year. So the reason that that matters in the context of this thing called the
interaction timeline is we get a lot of requests about updating tools and solving problems.
And one of the things at the time that we were dealing with and Wikimedia is still dealing with is a
well-known toxicity and harassment problem. Now if you're not an editor, you might not know about this,
but this is something a lot of editors deal with on a day-to-day basis is sort of growing toxicity
and say the community. So this is something we saw pop up again and again on the wishlist.
And so we were wondering, is there some kind of tool or intervention that we could engage with,
for example, of social networks like Reddit or Twitter, have a variety of what we call trust
and safety tools that help them analyze and spot patterns inside of things like toxicity.
And so we were thinking, is that something we can do? One of the things Wikimedia
has, so we don't have moderators, we have editors. We have special roles that editors can hold
called administrators. And those are actually the folks that when there is, let's say,
some kind of disagreement inside of Wikimedia, the administrators will step in.
Then there's a special committee called the arbitration committee when different kinds of
harassment cases sort of keep on going and ongoing. You have to bring in this very special group.
We like to sort of describe them as the Supreme Court of the Wikimedia of English language
Wikipedia. And the reason that that matters is so when I, when I worked at the Wikimedia
Foundation in 2018, we knew that, again, the harassment was this really big issue.
And we had seen sort of this request again for something to be, to be done about it.
So we sort of utilized requests from the community wish list as well as just conversation
that we had picked up from different Wikimedia conferences to sort of go back to the community
and say, this is something we're going to focus on. Is this something you think we should focus on?
And we got a lot of, like, yes. But then what do we focus on? This is where user research comes in.
So we did a variety of surveys and quality of interviews with all different kinds of community
members. So folks that ran specific meetups related to English English Wikipedia like women and
red, which is an initiative to get more female scientists inside of the Encyclopedia.
Also with folks that had faced high amounts of harassment, as well as different members of
administrators in the arbitration committee or our comp as we call it.
We ran these two surveys and also, again, held in-person focus groups and in, like, 101 interviews
at events. And so we had to sort of then present all this back to the community. All in all,
I think we worked with or spoke to over 400 contributors. This also includes our surveys. We weren't,
you know, one to one interviewing, 400 people. But again, this really matters. And one of the things
we kept hearing are one of the things we started to realize is that there might be a space for
making a tool to alleviate certain problems we were having. And tool changes and prototyping might
go a lot faster than us trying to change policy while we work towards changing or thinking about policy.
One of the things we kept hearing from these surveys and speaking to the administrators is this
thing called sock puppeting. So sock puppeting is a major harm. It's where you sort of pretend
to be a user, which you're not allowed to do. And so that was one thing that popped up that was
a big concern. This is something where let's say someone who works for a film production company
will write a page, like a wiki page about a movie. You're not allowed to do that. You're not allowed
to be paid to write wiki pages. So we'd also see people who had been banned from one of the,
like, like, from let's say English English Wikipedia coming back and creating a new account
and then going and engaging further forms of toxicity. So we knew that this was a major problem.
We also knew that something called wiki-hounding, which is digital stocking, was also a problem.
And so from our user research, we suddenly could isolate right these two major issues that we
had to do something about. This helped us provide, again, examples. Like, we could start creating
these anonymous examples to better understand harassment, also presented back to the community.
But from sock puppeting and wiki-hounding, the reason I bring those up is we started to realize
that there might actually be something we could use that could help our community administrators,
which is trying to visualize the activity of a sock puppet account and potentially visualize
the activity of someone engaging in wiki-hounding or digital stocking. Because a lot of what we learned
to the metrics of those accounts would be doing is they would be editing or changing edits,
or, again, writing and doing something to engage with another user. And all this actually is
trackable inside of the wiki-mediate projects. So one of the things the administrators mentioned
over and over again was not being sure of what a sock puppet account was doing of like how much
damage let's say they had done inside the encyclopedia, like how could they understand to build a
case to then say like maybe we should ban this account. Same with wiki-hounding. How much like how long
has this been going on? How do we know that it's happening? This sort of came up again and again
and again in our findings. And so we create this tool that I'll quickly show you now the
back to this slide. So we create this tool called the interaction timeline. And the way that we
also decide to create this tool is by engaging in ecosystem mapping to see what kinds of other tools
existed. And it turns out that an editor seven years prior to my team in 2018 starting this
head built a tool called the interaction timeline. But it didn't have a visual interface to it.
So all you could see was like line line line line line line line. And it hadn't been maintained.
So one of the things we thought was maybe this is something that could actually be helpful.
We created a series of wire frames. We tested with different members who are administrators
and are upon to see what they would think. And we also tested with high impact editors who
did not hold those roles. We actually got a lot of really great feedback. One thing I will say is we
design this for the wiki media community. So a big thing was can you have there be less space?
Can you have there be like less color, meaning like they like things very condensed and sort of
close together? We're like, yeah, we can do that. Why not? So then it's sort of going back to this.
This is the success of the tool was very action specific. It's one small tool inside of a
large ecosystem of problems. The goal is not like we cannot fix harassment with one tool and with the
action of like one piece of software. But it was something we knew that would alleviate the pain
administrators were facing. Because to report her harassment side of wiki media, there's not
a reporting tool. You go to a public page called admins no sports slash incidents and you have to
write what is happening. And admins don't have to go and track down what is happening between
two users. And this tool ended up has been very integral and those administrators to see what
happens. Because the administrators are the ones that weigh in and make a decision on any kind of action.
So if they ban someone from interacting with another user, they have to back up why? Because it's
wicked media. You have to cite your sources. So anyway, we're at time when we're going to quickly
say this really quickly. How did this sort of use both user research and ecosystem mapping?
The user research helped us realize we had a problem toxicity and helped us to refine
actually very specific nuances to that problem. A lot of this sort of here and see what
what data data administrators were facing and understanding some of the major issues they had,
which was admins no sports slash incidents. That was very toxic place. It was like the place that
they had to engage in. And what they needed was help in understanding the cases. And this is where
this tool came in. We also looked at other tools inside of the trust and safety world to see if
there was some other kind of tool we could use. So we also looked outside of outside of the
with media ecosystem. But we realized that a tool that an editor made was the best one.
And also for us, the reason that we also wanted to work on that tool maintained that tools
because it came from the community. It was designed for our community. And so we went and asked
that editor, do you mind if we update this? And he said yes of course. So then now it's something
that's updated and maintained by the foundation, by a specific team. So we know that it's continually
maintained and updated. And one of the things I'll say really quickly is, regardless of your project,
or product or community, it's always worth thinking about consistent ways for users to get feedback
and interact with you. And that's a big thing we had at Wikimedia. We had a very special page
for our team. We had a community liaison who sat on our team. So you could always leave comments
or email us. And we shared everything in real time, also on wiki. So people could provide
real feedback, also with a suggestion of please email us. We'd also set up one-on-one interviews.
So thank you. Look out for patients.