Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2019 Nov 1;26(11):1195-1202.
doi: 10.1093/jamia/ocz064.

Optimizing clinical trials recruitment via deep learning

Jelena Gligorijevic  1 , Djordje Gligorijevic  1 , Martin Pavlovski  1   2 , Elizabeth Milkovits  3 , Lucas Glass  3 , Kevin Grier  3 , Praveen Vankireddy  3 , Zoran Obradovic  1
Affiliations

Optimizing clinical trials recruitment via deep learning

Jelena Gligorijevic et al. J Am Med Inform Assoc. .

Abstract

Objective: Clinical trials, prospective research studies on human participants carried out by a distributed team of clinical investigators, play a crucial role in the development of new treatments in health care. This is a complex and expensive process where investigators aim to enroll volunteers with predetermined characteristics, administer treatment(s), and collect safety and efficacy data. Therefore, choosing top-enrolling investigators is essential for efficient clinical trial execution and is 1 of the primary drivers of drug development cost.

Materials and methods: To facilitate clinical trials optimization, we propose DeepMatch (DM), a novel approach that builds on top of advances in deep learning. DM is designed to learn from both investigator and trial-related heterogeneous data sources and rank investigators based on their expected enrollment performance on new clinical trials.

Results: Large-scale evaluation conducted on 2618 studies provides evidence that the proposed ranking-based framework improves the current state-of-the-art by up to 19% on ranking investigators and up to 10% on detecting top/bottom performers when recruiting investigators for new clinical trials.

Discussion: The extensive experimental section suggests that DM can provide substantial improvement over current industry standards in several regards: (1) the enrollment potential of the investigator list, (2) the time it takes to generate the list, and (3) data-informed decisions about new investigators.

Conclusion: Due to the great significance of the problem at hand, related research efforts are set to shift the paradigm of how investigators are chosen for clinical trials, thereby optimizing and automating them and reducing the cost of new therapies.

Keywords: clinical trials; deep learning; deep matching; electronic health records; pointwise ranking.

PubMed Disclaimer

Figures

Figure 1.
Figure 1.
The proposed system for matching clinical trials to investigators. Investigators (on the left) provide self-reported specialties, and their patient visit history may be available. Clinical trials (on the right) contain documents describing the treatment and purpose of the trial, primary indication (PI), primary therapeutic area (PTA), and desired population. The matched pairs of investigators and trials contain investigator performance reports from historical data used by the considered models to learn to rank investigators.
Figure 2.
Figure 2.
System overview. The system has several layers consisting of multiple components employed for carrying out separate tasks: 1. Data layer. The data sources are integrated to create separate views for investigators and studies. Word2vec is employed to learn vector representations of relevant medical concepts occurring in all free-form public study texts (clinicaltrials.gov). All contents of the study data view, along with the learned medical term representations, are passed as an additional input to the offline training layer. 2. Offline training. DeepMatch (DM) learns representations for the medical terms driven to predict the investigators’ enrollment scores. DM’s weights are then stored on a distributed file storage. 3. Online ranking. DM’s weights are loaded in this layer and ranking is performed for upcoming studies in an on-the-fly fashion. An upcoming study is passed to the matching component where it is matched against existing investigators from the investigator data view; their enrollment scores with regards to the given study are then predicted; finally, the top k most eligible investigators are returned by the retriever.
Figure 3.
Figure 3.
DeepMatch(DM) model architecture. DM takes 2 sets of inputs that correspond to features of an investigator and text of a clinical trial from universes of medical terms |V| and concepts |M|. An investigator’s features consist of a list of his/her clinical specialty areas represented as an li(2)x|V| matrix and summarized EHR data of investigators’ patients represented as li(2)x|V|. A clinical trial features include a public report for a trial (text) and its PI and PTA; these components are represented as ls(2)xV and ls(1)x|V|, respectively. The embeddings of both input sets V and M are obtained through a series of layers, including fully connected layers with rectified linear units and bi-directional LSTM layers which learn interactions between words and concepts in each input independently. The element-wise product is then calculated for all pairs of the learned embeddings and organized in a matching tensor. Finally, the constructed matching tensor is passed through a series of cross-convolutional and pooling operators to learn the investigator–trial enrollment scores.
Figure 4.
Figure 4.
Average NDCG@K for DM vs 5 alternatives across 159 test studies, for K in range of 2 to 10.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Friedman, Lawrence M., Curt Furberg, David L. DeMets, David M. Reboussin, and Christopher B. Granger. Fundamentals of clinical trials. Vol. 4. New York: Springer, 2010.
    1. Martin L, Hutchens M, Hawkins C, et al. How much do clinical trials cost? Nat Rev Drug Discov 2017; 166: 381–82. - PubMed
    1. Hurtado-Chong A, Joeris A, Hess D, et al. Improving site selection in clinical studies: a standardised, objective, multistep method and first experience results. BMJ Open 2017; 7: 7. - PMC - PubMed
    1. Mullard A. 2016 FDA drug approvals. Nat Rev Drug Discov 2017; 162: 73–6. - PubMed
    1. How Do CROs Choose Sites and Investigators for Their Clinical Trials? https://www.theclinicaltrialsguru.com/blog1/how-do-cros-choose-sites-and.... Accessed January 2019.

Publication types