With nearly 75,000 students attending its flagship Tempe campus, Arizona State University has seen record enrollment. It also prides itself in having one of the fastest-growing research efforts in the nation. Over the last ten years ASU has nearly doubled research expenditures and currently ranks in the top five for research expenditures among U.S. institutions without a medical school. Total research expenditures reached $905 million in fiscal year 2023 and they are in the process of launching a new medical school.
CaRCC had the pleasure of speaking with Douglas Jennewein, Executive Director of the Research Technology Office at Arizona State University to learn how he and his department support such a rapidly growing research enterprise. An edited version of the interview is below, with some sections paraphrased for length and clarity.
Tell us about your RCD program at Arizona State.
The unit I’m responsible for is called the ASU Research Technology Office. It is part of the ASU Knowledge Enterprise, which is the larger unit overseen by the Executive Vice President for Research. The Knowledge Enterprise encompasses all of the technology support people and all of research, including things like human subjects, software and IRB management.
The entire research technology organization has about 100 staff. The Research Technology Office includes the traditional research computing kinds of things supported by about 25 to 30 people. What we at ASU call Research Computing just includes the supercomputing or HPC part. That includes eight systems people, three students, and a portfolio manager kind of-person. And then there’s me overseeing all of that.
We also have a Research Data Management team, which is about half a dozen people that is a companion group. Research Software Engineering, which is a new effort. I think we only have one actual person in that role right now, but we are hiring more. And we have a special group that we call the Computational Research Accelerator that is three people right now. They do deep facilitation sorts of things like specialized hardware, new GPUs, Grace Hopper, NEC vector cards, quantum processing units, and things like NVIDIA Parabrick software – all of the really deep technical-level facilitation.
We have a Research Engagement team that’s led by Marisa Brazil. That team has three people right now and does training and meets with researchers. They provide outreach, facilitation, and pre-award support and do engagement with the Deans of Research and department directors. So, it’s both internal and external engagement. And we just hired some folks in regulated research and cybersecurity. We also have a staff member supporting our efforts in Controlled Unclassified Information and HIPAA-aligned computing.
It sounds like a pretty mature program. Is that how you would characterize it?
We have experienced a lot of growth in the last three or four years, so it is becoming a mature program. A lot of these positions are still pretty new. But we’re at the point now where we have most of the services that we want to offer.
You characterized your team as providing the supercomputing or HPC side of things. Are there other groups at ASU that provide the computational side?
Kind of. Central IT folks offer some commercial cloud services. That’s really more on the academic side of the house than the research side. Some of the larger schools, like the engineering school, do have dedicated IT support groups that are embedded. They operate departmental or lab-level systems, some of which are HPC. So we do have some folks in other units that do that – but mostly people come to us for research computing and data.
Do you partner with the IT organization for things like storage or networking?
Not exactly. The central IT folks certainly support us with wide area network connectivity, campus network connectivity, authentication, IAM, and things like that. But the storage services, VMs, private cloud, all of that is solely with us. We do work with them, for example, with GCP at Google Cloud. Some of the nuts and bolts of that system reside with central IT just because ASU has one Google account or presence.
You mentioned you provide some training and education services. Can you tell us about those?
We host many activities every month, typically about 50 training events per year. We have a class for research or advanced computing called Supercharge your Research that’s offered monthly. And we have a Beginner’s Guide, which is a little more frequent. And then we have more specific things like Linux shell or moving data with Globus, MATLAB, NVIDIA Parabricks, a lot of Python things. And we also host the Carpentries. ASU is a platinum sponsor of the Carpentries and we host the shell/Git/Python- related software carpenter.
How is your program funded?
This gets very complicated, but the shortest answer is we are hard funded. We do apply for and win grants that can offset those costs and let us do new things and spin up new services. I have a couple of grants and we’re part of Jetstream 2, so we have some salary support there. Some of the other teams have a few grants but we don’t have a lot of soft-funded positions. By and large, we’re centrally funded and enjoy strong central support. Most of our money comes F&A and a state budget line item allocation called Technology Research Infrastructure Fund (TRIF), which is a state of Arizona thing. So, state funded.
Do you charge for any of your services?
Yes, we do have some paid services and we have some free services. HPC, super computing, we have three clusters, two of them are free. The HIPAA aligned cluster has some costs associated with it, but the two open science clusters are free to use. You go in there and, and, and do whatever you want. The paid options on the cluster are the traditional condo model.
We have a few different types of nodes that you can buy through Dell. We have pre-configured systems, large memory GPU, and regular compute that people can buy. Our main growth model is through the condo model. We provide some central amount of computing, some number of compute nodes, GPU nodes, all that, and all the networking racks.
If they wish to, researchers can buy their own dedicated or mostly dedicated node on the system where they are granted priority access. If they’re not using it and some other person comes in, that person is guaranteed four hours before the paid person can boot them off. So, the paid person can always get access to their system in four hours or less.
On the storage side, we offer network attached storage and a tape archive. We do have some cloud storage options though, like Wasabi. And we have private VMs for the situations where HPC doesn’t make sense. We do have a VM farm that we sell different sizes of VMs out of. And co-location. If people just want to host a server with us, there’s a rack charge for that.
How do you keep your teams organized and communicating well?
We have team leads. Tickets and day-to-day workload for the supercomputing teams and systems aspects are managed by Rich Hull. He’s the portfolio manager for the HPC group. Marisa Brazil oversees engagement and research service delivery. Gil Speyer oversees the computational research accelerator. And Torey Battelle manages many aspects of what we call the Quantum Collaborative, a State of Arizona funded initiative for advancing quantum information science and technology research.
We don’t convene the entire group very often because it’s so large, but we do convene most of the computing and data researcher-facing folks on Thursdays for an hour. We also have a systems-focused meeting where we talk about things like patches or broken servers. And we have an engagement meeting where we talk about training, engagements, and any new faculty members who have accounts on our systems so we can plan to go learn about them. Or, we might talk about if there’s a large proposal being developed and we need to go and tell them all the great things about research computing. So, we have lots of sub-team meetings throughout the week..
A huge amount of our work also happens on Slack. We do quarterly maintenance on our systems and most of that is orchestrated via Slack and Zoom. We do have people on site in Tempe and in Phoenix at the different sites, but all of that happened on those two platforms. We also have a ticketing system and we’re in the process of moving from a heritage system to ServiceNow. We use the ticketing system for things like account requests, software installs,and even support for proposals and at any given time, we have between 100 and 200 tickets.
Our systems manager is a project manager as well. He manages things like system deployments as traditional work breakdown structure projects. For larger things that maybe expand beyond research technology we also leverage the knowledge enterprise project management office.
What are some of your near and medium term priorities?
Earlier this year ASU announced that it is launching a medical school and this is a pretty big change. It’s an engineering-focused medical school. So our support for secure and regulated research, especially HIPAA, is going to grow substantially. Growing our capabilities and expertise in that space is one of my priorities right now. We also want to figure out how to best tackle science gateways and data portals and provide support for that on campus. Our support there could be better, and that’s a service area we want to grow.
One of the more nuts and bolts things we’re working on right now is relaunching one of the supercomputers. We have two open science systems. Sol is new and it’s a top 500 system. And then we have the older Agave system that’s been around longer than I have been at ASU. So Agave is being rebuilt and we’re moving it to our newer data center in Phoenix. We’re physically moving the whole cluster, rebuilding it with the same software stack as Sol. It’s a big, big project.
What’s on your long term horizon?
Far off I think, we will be in a situation where our researcher-facing staff outnumbers our system-facing staff. When we are doing broader activity in that space and where the system space is mature and built up enough. Most of these systems I just mentioned are very new. When we really mature on the technical side of things we can emphasize and build out the researcher-facing pieces. I think if you look at the larger centers, that is what they do. And that’s where we want to be.
What helps you determine your priorities?
There are a few different things. There’s the executive leadership of the university. You have to align with their strategic goals and priorities. Then there are the strategic goals of the Vice President. That’s where things like our emphasis on support for the ASU medical school come from. And there’s the Research Computing Faculty Governance Board, a board of about a dozen faculty members and one student, that I meet with quarterly. They bring us things like concerns with the system. That’s why we moved to FairShare a few years ago. Folks were not able to work the way they wanted to work. The condo model also came out of feedback from that group. We also learned from them that they wanted storage services, so we rolled out the archival and the long-term storage services as a result of that feedback. So, the board is a big one.
But beyond that, communities like CASC or CARCC and seeing what our peers are doing is another piece that informs it.
What types of outreach and communications efforts do you have?
That’s a huge part of Marisa Brazil’s job, and I do some on the executive reporting side. For executive reporting we have dashboards and reports that show enabled research dollars and the number of faculty at ASU who use advanced computing systems, along with the amount of money they bring in.
We also have what we call investigator resources. That’s a webpage with proposal-support things like: how to get letters of support or collaboration; facilities equipment documents; and some data management or data storage pieces. And we have a short, self-service document we call the research computing overview, where people can pluck data for their proposals – things like our number of CPU cores and our number and types of staff.
In addition, we have a good relationship with the pre-award team called Research Advancement because we’re in the VPR’s office at ASU.There’s a lot of them because ASU is a big school. We tell them to look out for things like large data or collaborators at other schools where there might be data movement requirements. We tell them, “If you’re talking to folks in these proposals or if they’re coming through, they probably need to talk to us.” Having that good relationship with the pre-award folks has been valuable.
How do you connect with existing or new researchers to make them aware of what services you offer?
This is hard because ASU is a big school and we do enjoy significant support from researchers. We have around 3,000 users on the supercomputers, but there are many more thousands of people that perhaps we aren’t reaching or that we could reach better. And there’s, again, multiple prongs here. For one, I meet with the folks at the top. I meet with the deans or the directors of the large schools, the folks who are already big users of these kinds of systems such as engineering, arts and sciences, and health solutions. We have good relationships with them.
And Marisa meets with the research-focused folks in all of the units, including the vice president here in Knowledge Enterprise. And then through those relationships, we try to get into the department meetings and in front of researchers. I just did this with the natural sciences chairs and that came out of earlier meetings with their Dean and ADR. You want to get in front of faculty, and you don’t want to talk only to the deans or the chair, but it is sometimes easier to convey the value or the impact when you have their buy-in.
Do you have any success stories that you’re particularly proud of?
One that comes to mind is an NSF award to support the compact x-ray free CXFEL device, which is, I believe, a kind-of compact particle accelerator device, although I’m not a physicist. It fits in the basement of a large facility in one of our research buildings on the Tempe campus. This is a huge effort. It’s something like an $80 million award that we have worked on with the Biodesign Institute at ASU. We have been working since 2019 to get the infrastructure in place, especially networking, because of the huge amount of data. Network and data infrastructure was massive here. We had NSF on site during the pandemic and they asked us a lot of questions. We got through all the hurdles and the actual award was finally made early this year. So that was a huge success.
What do you see as your growth opportunities?
It was brought to light when we used the CaRCC Capabilities Model that we could stand to do some work in the software-facing area. We’ve tackled that first on the user application software, having folks who know how to build things that are perhaps different than the folks who configure Slurm or the InfiniBand network. But really, building user-facing scientific applications is its own job description, and we’ve got folks in that role now on the system side. But where I feel we need to grow, and are starting to grow, is really more in the research software engineering space, where folks are developing or better integrating applications for specific projects. In addition to that, we want to grow in the related area of web applications, science gateways, and, and data portals, working with tools like Globus and those types of software areas.
How did you personally get started working in the RCD profession?
I got involved in campus computing in the 90s when I was a student at the University of South Dakota. But in terms of scientific, or what we call now research computing, it was probably the mid 2000s. I started on the traditional IT side of things doing network administration, server administration, DNS, back when the internet was new and “internet” was a job. After finishing my master’s degree, I started playing around with what everybody back then called Beowulf clusters, and eventually decided to try to do something scientific with it.
Around 2004 I built (I’m not too proud to say), a Celeron cluster of desktop PCs that ran MPI Blast with the old www. blast interface for a researcher at the medical school. That got some attention and the medical school liked it. A few years later, I got some funding for an actual supercomputer, an AMD Opteron cluster of sixteen nodes – and that was the start of supercomputing at University of South Dakota. Back then, we used a software suite called Oscar and eventually pivoted over to Rocks. I think both of those are defunct now.
I did that in the 90s and early 2000s and then kind of made this pivot – really just out of a curiosity for what this Beowulf thing was all about – and wanting to do things with extra hardware that we wanted to breathe more life into. So that’s how I got into the HPC space, and then I met the folks at Arizona State in 2017 and eventually made the move, out here in 2019.
In closing, what’s your elevator pitch about your team?
The Research Technology Office is a Research First Technology Organization at ASU under the VPR. And our goal is to, as we say, give minutes back to science. We offer services like advanced computing, supercomputing, data storage, virtual machine server co location, training, facilitation, and data management.