-
Evaluation of software impact designed for biomedical research: Are we measuring what's meaningful?
Authors:
Awan Afiaz,
Andrey Ivanov,
John Chamberlin,
David Hanauer,
Candace Savonen,
Mary J Goldman,
Martin Morgan,
Michael Reich,
Alexander Getka,
Aaron Holmes,
Sarthak Pati,
Dan Knight,
Paul C. Boutros,
Spyridon Bakas,
J. Gregory Caporaso,
Guilherme Del Fiol,
Harry Hochheiser,
Brian Haas,
Patrick D. Schloss,
James A. Eddy,
Jake Albrecht,
Andrey Fedorov,
Levi Waldron,
Ava M. Hoffman,
Richard L. Bradshaw
, et al. (2 additional authors not shown)
Abstract:
Software is vital for the advancement of biology and medicine. Analysis of usage and impact metrics can help developers determine user and community engagement, justify additional funding, encourage additional use, identify unanticipated use cases, and help define improvement areas. However, there are challenges associated with these analyses including distorted or misleading metrics, as well as e…
▽ More
Software is vital for the advancement of biology and medicine. Analysis of usage and impact metrics can help developers determine user and community engagement, justify additional funding, encourage additional use, identify unanticipated use cases, and help define improvement areas. However, there are challenges associated with these analyses including distorted or misleading metrics, as well as ethical and security concerns. More attention to the nuances involved in capturing impact across the spectrum of biological software is needed. Furthermore, some tools may be especially beneficial to a small audience, yet may not have compelling typical usage metrics. We propose more general guidelines, as well as strategies for more specific types of software. We highlight outstanding issues regarding how communities measure or evaluate software impact. To get a deeper understanding of current practices for software evaluations, we performed a survey of participants in the Informatics Technology for Cancer Research (ITCR) program funded by the National Cancer Institute (NCI). We also investigated software among this community and others to assess how often infrastructure that supports such evaluations is implemented and how this impacts rates of papers describing usage of the software. We find that developers recognize the utility of analyzing software usage, but struggle to find the time or funding for such analyses. We also find that infrastructure such as social media presence, more in-depth documentation, the presence of software health metrics, and clear information on how to contact developers seem to be associated with increased usage rates. Our findings can help scientific software developers make the most out of evaluations of their software.
△ Less
Submitted 5 June, 2023;
originally announced June 2023.
-
The NLP Sandbox: an efficient model-to-data system to enable federated and unbiased evaluation of clinical NLP models
Authors:
Yao Yan,
Thomas Yu,
Kathleen Muenzen,
Sijia Liu,
Connor Boyle,
George Koslowski,
Jiaxin Zheng,
Nicholas Dobbins,
Clement Essien,
Hongfang Liu,
Larsson Omberg,
Meliha Yestigen,
Bradley Taylor,
James A Eddy,
Justin Guinney,
Sean Mooney,
Thomas Schaffter
Abstract:
Objective The evaluation of natural language processing (NLP) models for clinical text de-identification relies on the availability of clinical notes, which is often restricted due to privacy concerns. The NLP Sandbox is an approach for alleviating the lack of data and evaluation frameworks for NLP models by adopting a federated, model-to-data approach. This enables unbiased federated model evalua…
▽ More
Objective The evaluation of natural language processing (NLP) models for clinical text de-identification relies on the availability of clinical notes, which is often restricted due to privacy concerns. The NLP Sandbox is an approach for alleviating the lack of data and evaluation frameworks for NLP models by adopting a federated, model-to-data approach. This enables unbiased federated model evaluation without the need for sharing sensitive data from multiple institutions. Materials and Methods We leveraged the Synapse collaborative framework, containerization software, and OpenAPI generator to build the NLP Sandbox (nlpsandbox.io). We evaluated two state-of-the-art NLP de-identification focused annotation models, Philter and NeuroNER, using data from three institutions. We further validated model performance using data from an external validation site. Results We demonstrated the usefulness of the NLP Sandbox through de-identification clinical model evaluation. The external developer was able to incorporate their model into the NLP Sandbox template and provide user experience feedback. Discussion We demonstrated the feasibility of using the NLP Sandbox to conduct a multi-site evaluation of clinical text de-identification models without the sharing of data. Standardized model and data schemas enable smooth model transfer and implementation. To generalize the NLP Sandbox, work is required on the part of data owners and model developers to develop suitable and standardized schemas and to adapt their data or model to fit the schemas. Conclusions The NLP Sandbox lowers the barrier to utilizing clinical data for NLP model evaluation and facilitates federated, multi-site, unbiased evaluation of NLP models.
△ Less
Submitted 28 June, 2022;
originally announced June 2022.
-
Accomplishments in Genome-Scale In Silico Modeling for Industrial and Medical Biotechnology
Authors:
Caroline B. Milne,
Pan-Jun Kim,
James A. Eddy,
Nathan D. Price
Abstract:
Driven by advancements in high-throughput biological technologies and the growing number of sequenced genomes, the construction of in silico models at the genome scale has provided powerful tools to investigate a vast array of biological systems and applications. Here, we review comprehensively the uses of such models in industrial and medical biotechnology, including biofuel generation, food pr…
▽ More
Driven by advancements in high-throughput biological technologies and the growing number of sequenced genomes, the construction of in silico models at the genome scale has provided powerful tools to investigate a vast array of biological systems and applications. Here, we review comprehensively the uses of such models in industrial and medical biotechnology, including biofuel generation, food production, and drug development. While the use of in silico models is still in its early stages for delivering to industry, significant initial successes have been achieved. For the cases presented here, genome-scale models predict engineering strategies to enhance properties of interest in an organism or to inhibit harmful mechanisms of pathogens. Going forward, genome-scale in silico models promise to extend their application and analysis scope to become a transformative tool in biotechnology.
△ Less
Submitted 15 December, 2009;
originally announced December 2009.
