Thank you.
I'm going to try to speak loudly, which hopefully will be OK, and hopefully not too bad for people online as well.
So, yeah, so thanks for the opportunity to be here.
In this endeavor, I'm going to talk about research software engineering, kind of in one stream,
and then what's been going on at the University of Illinois or Banish campaign with regard to research software engineering kind of at the same time.
So, I guess first, just how many people know research software engineering?
OK, this is probably an unusual room here.
That's hard to tell.
So, wait, now I feel like maybe I should just skip the first six or seven slides.
Well, we'll try and see what happens.
So, maybe there will be something that you don't know.
OK, so first, I'm going to start off with basically just defining research software, which I think probably everybody here has some idea.
I don't know how many people have a formal definition.
And so, in this group that more end-grimbeder led with a few of us over a couple of years ending in 2021,
we tried to define this through a discussion.
And the definition that we came up with is that research software has a bunch of different forms,
but it's things that were created during the research process or for research purpose.
And there are other pieces of software that are used in research, like operating systems and libraries
and dependencies and package management systems and other things that are that weren't created for research,
and they're not really intended for research.
And so, we said that those should be called software in research and not research software.
And this varies between disciplines.
Sorry if you heard me say this yesterday, but if you are, I'll make a different example.
If you are a computational linguist, and you have some code that you've written in Fortran,
let's make this a really strange example.
You need a compiler in order to actually run that code.
And so, the compiler that you're using is not research software for you as a linguist.
But, on the other hand, if you're a computer scientist who is studying Fortran compilers,
maybe there's such a thing still.
That same compiler, if you would be research software.
So, it's a context question.
We can argue about this definition for a while, and in this process, we did argue about it for about 18 months.
And we came up with this as the best thing that we could agree on.
It's not necessarily the right thing.
It's not necessarily what any one person thought was the right thing, but it's what we got to.
And lately when I've been saying this in talks, people want to have this discussion again.
They say, can we have a community group that goes through and comes up with a better definition?
And my answer is, sure, let's try it and see.
I don't think we'll come up with a better definition, but if anybody wants to try it,
I think it's worth doing, and maybe we will.
Okay.
So, research software then typically comes from research.
This has happened from the start of computing, more or less.
Software started appearing around 1948 as best I know, somebody can correct me if they have a better definition of better term.
Research software started appearing in the early 1950s, just a few years later, and it was initially weather software as far as I can tell.
And then another roughly 20 years more or less until we got to software engineering and starting to think about how do we do this in a reasonably professional way as a discipline.
The people that are writing research software who are researchers, sometimes faculty, sometimes other people generally don't know good software practices still.
And we have software engineers who really do know good research, sorry, good software practices, but they don't understand the context of research.
And so you get situations where software engineer comes in and they say, please write down the requirements and then I will build software that meets those requirements.
You say, I don't know the requirements, this is research, and so there's that kind of question about how do we go back and forth between those things.
At universities and other places where research happens, you often do have students and postdocs who initially don't know good software practices, but they kind of pick them up over time during their career.
And they get to a point where they're actually fairly decent at this sometimes, and then they leave and they go somewhere else.
Some of them stay around and they become the staff at an institution and that's great if that happens.
Sometimes they're called a postdoc for a very long time and they're not officially ever called a staff member, but that's a different problem.
And and eventually we do get to a point where we have staff who have an understanding of research as well as software engineering skills.
And I think that's where we are today.
And so one one way of looking at this is the difference between the the crafts person and the scholar where we can say that a scholar is a researcher who's really driven to understand things.
They find intellectually demanding problems, they work on them until they really understand them and then they pass on that understanding through teaching.
And then we have crafts people who are driven to create and and leave behind an artifact that really.
That that really says what they understand that that is the understanding in some sense.
This is people that feel pain when they make something that's fragile or ugly.
They prefer to make things that explain themselves although they certainly often will understand the need for documentation as well.
And this is work that requires patience and pride in doing a job well.
And so scientific software research software needs people that can understand and do both of these things.
And I just want to point out this is from blog post from 2012.
So this is not a new idea at all. This is something that's been around for almost 15 years at this point.
Sorry.
So if we then think of a research software engineer and an effort to try to define this a little bit.
We have a space which I'm going to define with two axes because we're in a research context so I can do this.
We have a software engineering axis and we have a research axis.
And so we have people that are software engineers that are really great at software engineering.
We have people who are researchers that are great at research.
We have people who are researcher developers who are kind of somewhere in the middle that are quite good at research but also are decent at software.
And the idea is that we have research software engineers who are really more in the middle and can do both of these things.
And there's not a point to there's kind of a cloud of which we have research software engineers somewhere better at research somewhere better at software engineering
But they're pretty good at both of these things and that's kind of the key here
So in 2012 in the UK I believe in Oxford, but I'm not 100% sure if I'm right about that
There was a collaborations workshop and
Breakout group at this workshop was talking about this problem and what they realized was that there were a lot of people that were doing this work
That were doing research software with an understanding of research and trying to apply good software engineering practices
But they didn't have a common title and in the UK they had 194 different job titles that they found in job listings that were basically doing the same work
So they said let's come up with a common title. Let's call this research software engineer RSE
They decided that there wasn't any community of these people all these people with different jobs and different places different titles had
felt like they were alone often and so they created an association that's turned into a society and there's now
Others societies and associations and other places and they realized that this wasn't a profession and so they started working with their
Universities to develop career paths and a and a formal structure by which people could be hired and could get promotions and and
Could be evaluated based on their work in this in this job
So 13 in plus years later
We have again this this movement this term that was born in the UK
The UK RSE Association formed with about 50 people in 2013
It's now a professional society a non-profit do's paying
Organization with 700 official members and almost 7,000 unofficial members
And there's other associations in a lot of other countries at this point as well
Belgium Germany Netherlands Nordic countries Australian New Zealand Canada Chile
In the US we have a US RSE organization that has about 3,800 members now
So we're kind of growing a little bit behind the UK, but catching up hopefully
There's an RSE like group in France that's called devlog that actually started before the RSE movement and it has kind of
Its own history and does things all a little bit differently, but there's a lot of overlaps as well
And there's new associations that are forming or at least trying to form in in Africa and in Asia
But primarily I would say in South Africa and India at this point, but trying to expand beyond those as well
And these associations then can work on local issues collectively
So so in the US working on issues of despisic to the US
But then also we can coordinate when there's an effort to try to work together across countries to try to change culture to
Change practices at a higher level
Okay, so that's the RSE side of things
If I go then on to the side at the National Center for Super Computing Applications at the University of Illinois
What we've been doing
NCSA started in 1986 so 40 years ago actually like a week ago, or so it was our 40th anniversary
Professor Edil and I submitted a proposal without being requested to without submitting a two particular call to create a US
High Performance Computing HPC capability the next year, sorry in 1985 and
After a lot of discussion from the funding agency they decided to seem like a good idea and it's something they should do and they supported five national centers that were formed in NCSA. It was one of those five
We ended up then I would say catalyzing the internet and the web under this support
In the first year we created NCSA Telnet which was in some ways intended to help people connect to these super computers
We created HTTBD to serve websites so that people could then share the information that they were creating
We created Mosaic
a few years later, which was the first graphical web browser that I would say in a lot of ways kind of started started a lot of the
The web is
Well initially as a scientific activity, and then eventually into more of a commercial activity
And then we got to a point where there were some problems
So the core funding that we were getting from the funding agency ramped down over a few years
So we had this core funding that supported everybody for ten years and then five years later
We didn't have any of this core funding and so that led to a lot of challenges in terms of what do people do and how flexible are they and how do they work?
All of the funding became competitive so in order to get funding we had to write proposals and compete with people in every other university every other lab
Some large projects one some of these competitions as well as some small projects
But then some people left because they were unhappy with this competitive landscape and they were looking for other things and and we also wrote some proposals that didn't exceed and so then there wasn't money for everybody and some other people left
And we ended up at a situation where we had multiple groups in software that had overlapping skills
We had internal competition were different groups were creating projects that competed with each other
So we were developing three different workflow systems
We had internal competition on grants about who would be allowed to propose to particular things if there was a limit
Internal competition for collaborators right a good computational scientists in some department would be approached by multiple groups
The NCSA that wanted to collaborate with them and they didn't want to do that. They just wanted to collaborate with one person and do their research
Internal competition on staff when we wanted to hire somebody who would hire them and how that work
Huge discrepancies on titles and salaries because different groups were running under different systems and so that led to unhappy employees and
In equity and people that couldn't necessarily move from one group to another one project to another easily and
New clear no clear entry point for new collaborations as I said and finally no clear path for advancement except if you were a PI and you want to grant
That was a good thing and that would help you advance. Oh, and no job security as well because if you didn't win then that was it
Okay, so that was bad
But then things got a little bit better
Well, more than a little bit better
So one of the NCSA groups at that time was called
Image and spatial data analysis or I SDA and this was kind of an isolated group that was working in this particular area of image and data processing
And a person named Kenton McKennery who is one of the co-authors of this talk took over that group and
And he was frustrated with the situation and he wrote a manifesto that basically said how things should work in his opinion
And that included things that said basically the PI role as sacred
All right the PI's are the people that should be making decisions not some other level of management
There should be an emphasis on job security
Governance and decision making should be as democratic as possible among the PI's at one level and among the staff at another level
There should be an agreed upon hierarchy
They should be agreement between the PI's about when to hire and when to expand and who to hire
It shouldn't be each person doing it just for their project and there should be agreement on which projects to pursue
And that trust was the key for efficiency and that if the PI is could trust each other and could work together
Then they could actually collectively support all of the people and grow the organization
And so some new principles then came out of this and
The principles that we ended up with were to try to align projects and diversify funding
So the idea is that if you have a big project you use it to start small projects that do some of the pieces
And if you have small projects you can bring them together to turn them into bigger projects
And the bigger projects get bigger funding that's more sustainable the small projects have shorter term funding
But there ways of exploring ideas. So thinking of these all together as a system
I'm trying to reuse this software as much as possible
Trying to actually be a group as opposed to a set of projects. So the PI and staff have a larger voice
A louder voice together
To have a management committee that beats in addition to team meetings to bring everybody together again
To look for opportunities for synergies and ways of leveraging the work that's happening in one place and another place
And to provide stability for staff. So if one project ends, there are other projects that can hire the staff
Because they were involved in bringing them in initially and they know who they are and they know what they do
To think of ourselves as collaborators and not contractors
And I think this is actually a big deal that happens for a lot of research software engineers
But it was really I think a big change for us
So if we're working with somebody that's a researcher
We want to be seen as a as a copi. We want to be involved in the process of developing the ideas of doing the work
To be co-authors on the paper that comes out things like that not just to be somebody that you go to and say can you write some code for me that does this
And then finally and this is a little bit repetitive but a little different way of looking at it is
To think of staff members as working for the institution not for the project and they're hired by the institution
They're not hired by the project
That there's one broad shared reusable continuous job ad that is reused by any project that needs to hire people
We don't have to have each project right their own job ad and have their own competitions
That there's a rotating hiring committee
that helps
Spread the load across people that we have a formal career path and we use senior staff to do training in a formal way
as well
so
Vision and goals for NCS and software
So one of them is that we support scientific discovery through software and this is really research discovery
Sorry, we always have issues with using science versus research, but it's it's really scholarly research
We amplify the efforts we support again. We're not just writing code. We're actually making the project better
We're known for our staff and our software and we're a leader in the research software engineering movement
We strive to lead in the areas that we undertake and that we embody trust and responsibility
So again, these are principles that we try to work with with different researchers. We don't always do all of these as well as we should but this is what we try to do
So if we look then back at the the RSE movement and how we're fitting into it
So I joined NCS a in 2016 about 10 years ago and I was active in the RSE movement
Through the UK at that point and so so I brought this idea and I said you know
I think this is what we really are right. We are RSEs. We're not just
Research developers. I think was the title that was being used before that
And so this gave us a sense that we at NCS a weren't alone. There were other people that had the same problems and we could learn from them and work with them
In 2018 I co-founded the US RSE organization and I was on the steering committee for the first four years
And then Kenton McKenry from again from NCS a was on for the next three years and now we have a pinging shen
Who is also on this so we're trying to make sure that we have a continual role in shaping the organization
At least as the members vote for us
And we see ourselves as a US exemplar of an RSE group and so I just want to point out one one paper that we wrote in
2019 that was looking at the model that we had the model that was taking place at the University of Notre Dame and
The model that was taking place at the University of Manchester in the UK and so we compared these three different models for how do these different groups work?
How do they hire people? How do they find projects? How do they support themselves and and we think that this is I don't know three out of maybe
I don't know five eight different models that maybe makes sense maybe there's more but but we think people that want to start a new group
Can look at these models and see what other people have done
We also have again a formal career path. I know you can't read this
Sorry, let me just jump back for this second and say the slides are at this DOI
They're also linked from the fussed-end page about this talk so you can you can look at any of these at any point
But we do have a right a formal structure where we are
Things like
Responsibilities that change over five levels from associate research programmer all the way up to principal research programmer
Decision making that comes in a different point super vision that comes in a different points
Kind of requirements about education and training and experience and then kind of preferred experience and so we've tried to again formalize this
So that when people want a promotion
We know what they need to be doing at the next level and we can be we can see are they able to do that and they can see what do they need to learn as well
And and this gives people also of course the the chance to move up over time
So I'm getting close to the end just to mention
We kind of have three different formal project roles right now
We have the the PI or copi or the Stuart the person that's responsible for this excessive of a project
Their job is
Primarily oversight and guidance and organization and and helping staffing and and being the person that's responsible for the deliverables
This is generally some of that's working like about 8% of the time
So this is kind of like one month a year spread out over a year
And these people can work on multiple projects at once
And we have technical leads who are the person that's really responsible for overseeing development
Coordinating the developers
Probably doing some of the development themselves but also training the developers the newer people the younger people that are coming in
And this is often about a 10% position and again they can be doing multiple of these at the same time
And then we have the core developers who are actually doing the the real main development work
Perhaps they're involved in publications and reports as well as they get somewhat more senior
Ideally, this is 50% time they're working on two of these at a time
In practice we don't always get 50% of people from a project and so sometimes they can be working on maybe up to four for more senior things
So maybe four projects at a time we try to keep junior people more at two
But again, as people go up maybe it's a little more. I'm I will say
50% time my deal the reason we say that is that if we get full-time support for somebody
We hire two people at half-time
All right, we don't hire one person
And the reason for that is so that there's redundancy so that there's backup so if somebody leaves somebody else can come in
But also so two people can talk to each other about the project and it's not somebody just working on this in isolation
So that's a kind of a conscious choice that we've made over time
Okay, so next the last slide is just an archite of where we are so there is a few people that are kind of up at the top leading
We've got a bunch of different groups that are focused in different areas and
Yeah, and I think things are are generally going well when we are able to build different skills in different areas that help with
What our collaborators are interested in?
So to wrap up
We've had a bunch of successes. I think at this point
This software director is about a fifth of the organization in terms of both staff and funding and I think that's probably a reasonably healthy healthy size
We haven't had anybody that had to leave because of lack of funding in at least 10 years and I'm not sure if anything happened before that
Because that's the time that I've been there. I mean obviously people have left for a variety of reasons
But but nobody's had to leave because we didn't have the funding for them and and I think that's
quite important in in telling people that this really is an
An institution that they're joining and not a not a three-year project. They're joining for example
We have I think quite a happy staff and in a good culture and people see opportunities for personal and technical growth
And I think we're making an impact on research at Illinois in the country and on the RSE movement
We do have a couple of challenges though
So one of them is funding
The current external environment is a quite a large challenge. I don't know if you've heard but there's some stuff going on in the U.S
And it's made this a little bit difficult
So we're not exactly sure for a variety of reasons how much funding we're gonna be getting in from different government sources
and philanthropic sources and industry as well and so there's lots of different effects that are leading to challenges
We also have almost no institutional support from the university as a whole in terms of funding
And there's other RSE groups that are very heavily supported by their institutions. All right, so we're supported by projects
We just have a lot of projects and we can kind of build up critical mass enough that this works
But if it stops working then we don't have the fallback of the institution. So we'll see what happens with that
And then the other thing is that hiring and AI are both making our existing processes difficult
So hiring is becoming more challenging recently and AI is
Making everything more difficult. I guess is a short way of saying it as we've heard in previous talks
So just as an example
We used to have a pre-screening for hiring or we'd ask people to do a programming exercise
Which doesn't make any sense anymore because they just ask an AI agent what the answer is
And so we don't know how to evaluate people in a good way until we actually know them until we meet them and talk with them
And
AI as a whole also leads to questions about
With our collaborators
If they think that they can just use AI to write software themselves and they're going to write good software that's sustainable
Then do they need us and and how do we convince them that they actually still do and that there's something that we're doing that's different
So so we do have some challenges, but that's where all these things and
Happy to answer any questions. Thank you
Thank you
Couple minutes
Yeah, go ahead
If there's a sort of adjunct of the computer size department, that's kind of worked or a research as a cheaper
But they also
Yeah, so the so the question is kind of basically where where do RSEs fit within the organization and there's kind of pros and cons and different
aspects and so I think there's a few different models as you mentioned
In some cases they report to the the IT organization
In some cases they report to the the the research organization when there is a central research organization
Which is the case at Illinois at some cases they're more related to computer science as you said
It seems that there is not an answer for this at all and it really depends almost entirely on the institution and
And even when it happens when something happens in an institution
It's not necessarily the thing that you might say from the outside is best, but for some reason the people in the institution have thought it's the way that it's going to work
So it's
There is a lot of discussion about this and it I think it's another thing where there's a small number of models and they have pros and cons
But it's but it almost more depends locally on what makes sense rather than these kind of global questions
One more
question. Yeah, good
All down. Sorry. Yeah, sorry. Sorry. It's all you first. Yeah, yeah, but hiring would record open source
Commission away
And follow up
Yeah, because open source was mentioned when they took
Yeah, yeah, a property of your project. Yeah, yeah, yeah, yeah, okay, so
Let's see so
Is open source work a something that kind of can be brought in in the hiring process and can be
Evidence of somebody's qualifications and the answer is completely yes and
Ideally that happens and we're very happy when that happens, but sometimes we have people that are coming from industry
And they can't actually share any code or or something else. So it's
We like to do that when we can but it isn't always an answer and then and then a comment that's completely true that I didn't really mention open source
Which is kind of a strange thing here
But but I will say that from from my point of view, I think there's there's research software kind of as one space and there's open source is another space and they
They overlap a lot, but not completely
And and I think we would fall into that in which
Probably 90% I would say of what we do is open source
Not everything there there's some research software that we work on that's not but most of it is and and that's what we try to do and
And once somebody is add in CSA that is something that we
That we see as
Something to strive for is is working openly and and when we have projects that are open
We start we try to start them as open from the very beginning and so we we try to follow the best open source practices we can
With the recognition again that some projects that's not the right answer for even though almost all the time it is there's there's a couple where it's not
Did you want to
I assume that in your wallet the RSC doesn't have control over
The architecture they have any and what about do so yeah, so question about the RSEs and control over their own research
So the this is actually a tricky question because we have another track the besides the RSE track that's called a research scientist
And those are people that do have complete control over their own research
But have a much different model and they get evaluated based on papers more often rather than software
Although it can be both where the RSEs are getting evaluated almost entirely on software
We don't have a great answer for people that are in the middle other than as people get to be more senior and they can bring in their own projects or create their own projects
Then that big they have the flexibility to do what they want with those projects. So in some ways
If you can bring in funding to support something then you can do that as long as it doesn't seem like it's tremendously out of line with everything else
As long as it lines up with the institutions
Objectives
So if somebody wants to do some research that's related to software and they can get funding for it
They will have time to do that
But we're it's it's hard to start that up because we don't really have kind of seed funding or a way to to do that
So it almost has to to come out of a collaboration with somebody else or it has to be something that's such a great idea
That you can write a proposal without having done much work
And it happens
But it typically is people that are more senior that have had some experiences that they can build on and and turn that then into a proposal for an idea
Okay one one more. I'm not sure where we are in time
Okay, so I go ahead. Sorry
From this
Question is basically kind of how
There's an appearance that we're not interacting with the larger open source community
And they open let's say open data community. I'm not sure if that's exactly fair
But let's let me try that one for the minute
So we I think we do interact with them, but we don't necessarily
Talk about it very much. We kind of do it more as individuals as opposed to as an organization
So a lot of the people that are working on these projects are using other open source projects as part of them
And they're contributing back to those often as individual contributors and and that the time that they're using to do that
is time that the project effectively is paying them for
It's not kind of volunteer effort that they're doing on the side. So so I feel like we do
Participate in a number of projects in some cases some of the things that we do we try to turn them into larger or more general open source projects
We're often not as successful with that as we might like and and that could be from lack of
experience
It could be because things that we think are general are still somewhat specialized
I don't have a good answer, but we I would say that we do try to do this, but whether or not we're as successful as we could be probably as a question
Yeah, no, I
I mean I I appreciate the the comment and again I think as individuals we do this well as an organization
We probably don't prioritize it very much. Sure. We don't talk about it at least as enough
Yeah, okay. Thank you