DNAdigest readers, this week I am happy to present you to the Centre for Open Science (COS). Our interviewee is Tim Errington, who is the project manager of the Reproducibility Project: Cancer Biology, part of COS. Take a look at what exactly this non-profit technology company is all about and how it will improve research practices.
Tim Errington, Project Manager of the Reproducibility Project: Cancer Biology
1. You are a part of the Centre for Open Science team. Please, tell us what the mission and the interests of the company are?
Our mission is to increase the openness, integrity, and reproducibility of scientific research. We support shifting incentives and practices to align more closely with scientific values. We work on this through three main activities: metascience activities to research scientific practices, community activities to foster open science, and providing infrastructure support through the Open Science Framework (OSF).
2. Why do you think OPENNESS, INTEGRITY and REPRODUCIBILITY of scientific research are of such a great importance for the community?
Openness and reproducibility are core scientific values. They refer to increasing transparency of research data, methods, materials, and workflow, and to increase inclusivity so that everyone has opportunity to access and contribute to research based on their interests, skills, and availability. As we continue to build off of previous results, it’s vital that those results are shared. Knowledge accumulates by sharing information and through independently reproducing results, but without these principals we are left to believe in conclusions without ourselves exploring the underlying parts that went into it. In principle, anyone should be able to evaluate the basis of claims, and reproduce the approach to verify the claims. These values are central to the progress of science and vital for enabling findings to be effectively built upon by the scientific community.
3. Would you let us know what your part of the company is and why you personally decided to support it?
I am a Project Manager for our metascience efforts at COS and was drawn to the mission here because of the benefits of improving our research practices. I think I had some of the same experience all too many scientists have had – you read about an exciting result and want to repeat it and build upon it only to find the methods sparse and the data lacking. I had this occur to me when I first joined my PhD lab from a paper produced by my lab a year earlier. It took months of digging through old lab notebooks, talking with previous students, and trying slight modifications of the protocol details before I identified the necessary missing detail to reproduce the result. I thought if it was this hard for me to reproduce this result in the same lab that originally generated it, how would it be possible for someone else in another lab. I think we should always be looking for ways to improve our research process so that we as a scientific community can efficiently reproduce and build upon others results. This is aligned with the mission at COS and why I support and enjoy working here.
4. The COS has two main services so far: the Statistical & Methodological Consulting and the Open Science Framework. Can you let us know exactly how they help scientists reproduce their projects?
The OSF is a free, open source web application that is the central infrastructure for COS to support open science objectives across disciplines. Researchers use OSF to collaborate, document, archive, share, register, and search for research projects, materials, and data. Also, the OSF is a mechanism for connecting to other tools that researchers use, such as Dropbox, GitHub, and Figshare. Additionally, the OSF supports private and public workflows. By supporting the entire research workflow the OSF ensures all the aspects of any given experiment is recorded, preserved, and discoverable by everyone to explore and reuse. Documentation across the lifecycle will help evaluate the quality of claims and promotes reproducibility.
The statistical and methodological consulting that is offered through COS allows scientists to improve the replicability of their own work through documentation, adherence to standards, and the use of open tools, such as the statistical computing software R and the OSF. The free consulting can occur one-on-one through email or online video meetings, or through online workshops on topics related to reproducible research and good statistical practices. Additionally, on-site workshops for labs, departments, or organizations can be arranged. These consultations help researchers integrate reproducible practices and techniques into their current workflow, discuss the choice, application, and interpretation of statistical methods, and introduce tools to aid in making their research more transparent and reproducible.
COS also offers a free presentation sharing service hosted on the OSF. This is a way for posters and talks at scientific meetings to be hosted and archived for others to discover. Additionally, it allows the researcher the opportunity to add related content to the poster or talk.
5. Would you tell us more about the aim of Cancer Reproducibility Project, how does it work in practice and who is organising it?
The primary aim of the Reproducibility Project: Cancer Biology is to test the reproducibility of preclinical cancer biology results. The second goal is to evaluate the predictors of reproducibility. That is what factors are associated with (or not) replication. Like almost all experimental research, the interest is in using this particular sample of studies to learn something about cancer biology methodology in general. Papers were selected from the years 2010-2012 with a selection of key experiments included in the replication attempt. Additionally details can be found on the Reproducibility Project: Cancer Biology project page on the OSF and our feature article introducing the project in eLife.
The project is in collaboration with COS and Science Exchange with the results published in the open-access journal eLife. For each paper, the original authors are contacted to seek additional experimental information, including protocol details, data underlying the figures reported, and if necessary unique materials generated by the lab. Once this information is obtained, research labs from the Science Exchange network are identified that have the technical expertise necessary to conduct the experiments. Any known differences from the original experimental work are recorded with additional quality control steps included when possible to ensure the integrity of the replication attempt. The sample size and statistical analysis to be performed on the replication experiments are determined priori to ensure the replication attempts are highly powered to detect the originally reported effect. These replication plans are not only shared with the original authors for informal review and input, but also formally through a new publishing model, called Registered Reports, which eLife has adopted for this project. This approach has the protocol details and proposed analysis plan peer reviewed prior to the start of experimental work. All accepted Registered Reports can be viewed on the Reproducibility Project: Cancer Biology landing page on eLife with additional related content available each studies project page on the OSF.
After the experimental work is completed, all related content, including the raw data and analysis, will be uploaded to the project page on the OSF and a final Replication Study will be published in eLife. This project is an opportunity to demonstrate how open, transparent methodology can be accomplished in cancer biology. The resulting open methodology and dataset will provide evidence about the reproducibility of preclinical cancer biology results, and an opportunity to identify predictors of reproducibility.
6. How can someone reading this interview help your project?
Earlier in the project we had a community of volunteers, largely composed of postdocs in the life sciences, contribute to the project by extracting information from the original papers and drafting protocols for the replication experiments. While we are past this stage of the project, if anyone is interested in contributing, they can contact myself at tim@cos.io to explore possibilities. Additionally, any comments or suggestions by the community on this project or other reproducibility projects in other disciplines or sub-disciplines are welcome.
7. What are your hopes for scientific reproducibility in the future?
I hope to see a shift in existing reward structure of science to one that encourages and incentivises open practices and improves reproducibility. I hope to see a change in scientific reporting to not just allow, but to also encourage publication of replications, regardless of the results. And I hope to see scientific research and reporting become more transparent in regards to recording research details. As a final comment, I think scientific reproducibility can be improved with technology to enable change, training to enact change, and the incentives to embrace change.
Are you part of a project that facilitates data sharing for genomics research? Would you like to be featured on our blog? We would love to hear from you. Drop us an email at info@dnadigest.org or use our contact page to get in touch.
The post DNAdigest interviews COS appeared first on DNAdigest.org.