Optimizing clinical trials recruitment via deep learning

Data

The data used in this study contain information related to both investigators and studies collected over the past 20 years. The following provides a brief description of data collected and aggregated from several heterogeneous data sources:

• Investigator performance data. A proprietary data set collected by the CRO that bridges investigators with their history of performed clinical studies. All major biopharma companies and CROs maintain similar proprietary data sets. Therefore, this data set contains information on both investigators and the studies they participated in. The former relates to investigators’ areas of specialty, while the latter holds a textual format and includes information about a study’s primary indication and therapeutic area. In addition, for each investigator, the number of patients that he or she enrolled to each of their past studies is also provided. Consequently, since each investigator–study pair has an enrollment score, if a record of a given investigator is not present in this file, such an investigator has no enrollment history (ie, has never participated in a clinical study).

• EHR data. An integrated electronic health record (EHR) database of patients’ medical claims and prescription history that covers 65% of all physician claims and 92% of all prescription claims in the US. These records include information about patients’ visits over time in terms of the diseases they were diagnosed with (up to 15 per visit), procedures they underwent (up to 15 per visit), and the medications that were prescribed by investigators (up to 5 per visit).

• Public study data. Detailed study-related free-form text data crawled from clinicaltrials.gov—the largest registry of clinical studies throughout the world.

Upon collection, the data from the described sources were integrated into 2 separate views:

• Investigator data view. To create this view, investigators were retrieved from the investigator performance data. Thereafter, each investigator was represented by a vector of most frequent codes of diagnosed diseases, conducted procedures, and prescribed medications during all visits to the corresponding investigator stemming from the EHR data source.

• Study data view. Each record in this view includes the primary indication and therapeutic area of a single study stemming from the investigator performance data and further textual description of the study obtained from clinicaltrials.gov.

For a complete system overview, refer to Figure 2.

Building blocks

Medical concepts embedding layer. From the EHR data we chose the most frequently diagnosed, procedures, and medications prescribed by each investigator, resulting in an input of length $l_i^{\left(1\right)}=130$. $\mathrm{In}\mathrm{}\mathrm{particular},l_i^{\left(1\right)}$ thresholds from the most frequent 50 diagnoses, 50 procedures, and 30 prescriptions for each investigator, where threshold values are chosen based on basic descriptive statistics of medical concept codes across all investigators. An embedding lookup layer is built to learn distributed representations of medical concepts (having a dimensionality of $d_m=200$), after which 2 fully connected (FC) layers with ReLU nonlinearities are used to learn higher-order interactions between these embeddings, thus capturing complex relations between diseases, procedures, and medications. The final representation of this data source has dimensionality $l_i^{\left(1\right)}\times d_p$, where $d_p=50$ is the dimension of a joint representation in the same space with all input data components.

Medical and trial terms embedding layer. Investigators’ specialties, public trials text, and trials PIs/PTAs have a common vocabulary of medical terms. Thus, the inputs for the investigators’ specialties ($l_i^{\left(2\right)}=10$), trials PIs/PTAs ($l_s^{\left(2\right)}=10$), and public trials text ($l_s^{\left(1\right)}=100$) were embedded using a medical term embedding layer with a dimensionality of $d_w=300$. The investigators’ specialties and the trials PIs/PTAs are passed through 2 fully connected layers, respectively, building dense representation vectors of medical term interactions, each of size ${1\times d}_p$. As public trials text contains multiple elements concatenated into a single document, it is worthwhile to learn relations of terms in different segments. Thus, we pass the obtained representation of public trials text into a bi-long short-term memory¹³ layer such that the model can learn complex relations between terms (rather than single directional relations). The final representation of the public trial data become ($l_s^{\left(1\right)}{\times d}_p)$-dimensional.

Investigators and clinical trials matching tensor. The investigator–trial matching is obtained in this layer. Using the learned higher-order representations of investigators $H_i$ of shape ($1+l_i^{\left(1\right)}){\times d}_p$ and clinical trials $H_s$of shape ($1+l_s^{\left(1\right)}){\times d}_p$, a tensor is built for their implicit matching. A matching tensor $H_{\mathit{is}}$of shape $(1+l_i^{\left(1\right)}){\times \left(1+l_s^{\left(1\right)}\right)\times d}_p$ is created in the following manner:

where $m,n\in \mathbb{N},\mathbb{}1\le m\le \left(1+l_i^{\left(1\right)}\right),1\le n\le \left(1+l_s^{\left(1\right)}\right)$ and $\odot$ represents the element-wise product of $H_i(m,:)$ and $H_s(n,:)$. The first 2 dimensions of the tensor $H_{\mathit{is}}$are $\left(1+l_i^{\left(1\right)}\right)$and $\left(1+l_s^{\left(1\right)}\right)$, respectively. Here, each investigator’s medical concept is matched to each trial-related term, and the element-wise product of their vectors represents a $d_p$-dimensional thread in the matching tensor. With this operation, we aim to capture intra-relations between investigators’ medical concepts along with their specialties and clinical trials medical terms.

Learning to predict enrollment scores from matched representations. The matching tensor $H_{\mathit{is}}$from the previous block is convolved through the entire depth $d_p=50$ by 3 convolutional blocks with different filter sizes: $3\times 3,3\times 4,3\times 5$. The choice of this combination was determined by filter size fine-tuning. The number of filters is fixed to 6 for both the first set of convolution blocks and the final convolutional layer. Such cross-convolutional operators were successfully applied in the past to tasks of modeling similarities between general sentences.²² Interaction representations between medical concepts and trial-related terms are learned here, and they are passed through the ReLU layer, after which another $1\times 1$ convolution with ReLU is used prior to the 2-dimensional max-pool layer that embeds a whole investigator–study pair into a single vector. Finally, this vector is fed into a fully connected layer and passed through a squared-loss ($L={(y-f(x\left)\right)}^2$, where $f\left(x\right)$ is the network described above) layer to predict enrollment scores for investigator–study pairs.

The enrollment scores for each investigator per clinical trial are finally used for obtaining the investigator rank.

Experimental setup

The experiments were conducted in a manner that reflects a real-world situation from the clinical trial business such that the same trial–investigator pairs cannot appear in the training, validation, and test data subsets. In particular, all models were tested on studies that started in 2016 and 2017. Studies that started in 2015 were used for validation and hyperparameter tuning, whereas all studies that started prior to 2015 were used for training. The characterization of the training, validation, and test set sizes is given in Table 1.

The task is to rank investigators by their expected enrollment.

Baseline methods

The accuracy and relevance of enrollment scores predicted by DM was compared against the following alternatives (representing both current industry standard and state-of-the-art learning approaches):

• Median enrollment (ME). The current industry standard is to take the median enrollment from within the therapeutic area of interest to predict future enrollment scores.

• Point-wise support vector regression (SVR). A support vector regression model²³ that approaches the investigator-to-study matching in a pointwise ranking fashion.

• Linear model (LM). A linear model is employed for predicting investigators’ enrollment scores based on their handcrafted features (their specialties, as well as the frequencies of their patients’ diagnoses, procedures, and medications) and study text data (summarized by summing up the numerical vectors of clinical trial terms).

• Multilayer perceptron (MLP). A 2-layer MLP model is used to learn the second-order interactions of the handcrafted features and the summarized trials text to predict the investigators’ enrollment scores.

• DeepConcat (DC). A model similar to the proposed DeepMatch; however, in DC, trials and investigators’ feature maps are summarized into respective vectors which are only concatenated afterwards.

Evaluation measures. As our main goal is to rank investigators for upcoming studies, all models were evaluated using the normalized discounted cumulative gain (NDCG).^25–26 Given a list of investigator–study pairs ranked by their enrollment scores, NDCG is calculated as $\mathit{NDCG}@\mathit{K}=\frac{\mathit{DCG}@\mathit{K}}{\mathit{IDCG}@\mathit{K}}$, where DCG@K = ${\sum }_{\mathit{k}=1}^{\mathit{K}}\frac{2^{{\mathit{rel}}_{\mathit{k}}}-1}{{\mathit{log}}_2(\mathit{k}+1)}$ with ${\mathit{rel}}_{\mathit{k}}$being a tier relevance value for the enrollment score of the k-th investigator, whereas IDCG@Krepresents the ideal DCG (producing the maximum possible DCG through position K). The final NDCG values are reported as the average of NDCG values over the 159 studies in the test set (see Table 1). In the conducted experiments, in addition to NDCG@K, NDCG@N_i was measured as well, where N_i is the number of investigators that participated in the i-th study, and the average over all studies is reported as NDCG@N.

As most of the baselines directly predict investigators’ enrollment scores, we also compare these models using the mean squared error (MSE) measure.

Finally, to evaluate the models’ performance in a more domain-savvy manner, we measure their classification accuracy for the task of detecting the bottom/top 30% performing investigators, where the goal is to simply classify whether an investigator is within the top or bottom 30% of the performers on a given study.