NSF Prize will support a project to promote reproducibility in computing

With the support of a three-year, $900,000 grant from the National Science Foundation (NSF), Assistant Professor of Computer Science and Engineering Carlos Maltzahn and the Center for Free Software Research at UC Santa Cruz (CROSS) will engage in collaborative research to increase the reproducibility of computational research.

This grant comes from the inaugural year of an NSF initiative, called Findable Accessible Interoperable Reusable (FAIR) Open Science Research Coordination Networks (FAIROS MRC), to create groups of researchers who lead by example to promote the results and artefacts of open science. Overall, 10 new project groups have been funded to pool $12.5 million in building open source communities, which foster dynamic exchange of artifacts within a common infrastructure.

“There’s a huge shift going on,” said UCSC CROSS director Maltzahn. “I think it has to do with realizing the value that industry places on open source, and that open science and networks of expertise need to be more inclusive and involve university stakeholders, industry, government, and open source communities. This becomes especially important when you’re talking about revitalizing high-tech manufacturing in the United States.

Maltzahn will work with the Repeto Project, a group focused on practical reproducibility in computer science research. Reproducibility allows researchers to verify results, speed up the research process to gain insights faster, and see their products more widely used in research labs, classrooms, and industry. It also helps students better understand the thought process of the original researcher.

Involving researchers from the University of Chicago and New York University, the Repeto project strives to make reproducible computing practical – where many experiments can be repeated in a cost-effective way. Overall, they will create infrastructure, teach and mentor students, facilitate workshops, and create community best practices related to this goal.

Through these efforts, Maltzahn and his collaborators hope to better understand and foster the “reproducibility market” to ensure that artifacts, such as software, are available for replication, but that those artifacts are both useful and used.

“The goal of Repeto is to make creating reproducible artifacts really easy and consuming those artifacts really easy,” Maltzahn said. “The general idea is that practical reproducibility artifacts will speed up the research cycle, so you’ll get a much faster succession of information and a powerful toolkit to improve student training.”

UCSC’s role in this project will be to convene a global program in 2023 called “the summer of reproducibility”, following the model of CROSS Open source search experience program, which pairs students with mentors working on open source projects. Similarly, for the Repeto project, undergraduate students participating in the Reproducibility Summer will work to replicate published research.

This will allow students to gain a better understanding of the experiments they have repeated than just reading about them, and will allow mentors to better understand what is needed to make their work truly repeatable.

Maltzahn will work with UCSC CROSS Deputy Director Stephanie Lieggi to organize the Summer of Reproducibility. He will collaborate with lead researcher Kate Keahey, senior computer scientist at Argonne National Lab and senior CASE scientist affiliated with the Department of Computer Science at the University of Chicago, Haryadi Gunawi, associate professor of computer science at the University of Chicago, and Fraida Fund , Research Assistant Professor of Electrical and Computer Engineering at NYU.

University of Chicago researchers will focus on building and maintaining the infrastructure to make practical reproducibility possible. They will also organize workshops on topics related to reproducibility. NYU will focus on best practices for teaching and applying reproducibility in the classroom.

Thanks to the connections made by researchers interested in reproducibility within the framework of the Association for Computer Machinery Emerging interest group for reproducibility effort, Maltzahn and his team created an international steering committee for the project, involving people from all disciplines, including computer science, librarianship and social science.

All 10 groups of FAIROS RCN winners are expected to work together to share artifacts and will have monthly meetings led by the program director. The UC San Diego Supercomputer Center (SDSC) is also part of the cohort, making the UC system a major participant in this initiative. Cohort members such as North Carolina Central University provide exciting outreach opportunities for students and faculty at historically black colleges and universities (HBCUs) and minority-serving institutions (MCIs).

Previous KDDI apologizes for yet another network service outage
Next After denying red flag posts, PNP cries foul of 'fake news' on social media causing solved cases to resurface