Predicting UAV Type: An Exploration of Sampling and Data Augmentation for Time Series Classification

Most of the limited research on machine learning with UAVs is focused on anomaly detection, and thus is limited to smaller datasets that include coded anomalies. Park et al. [8] used an autoencoder to determine if a flight is faulty based on whether a threshold is met when reconstructing the vector representations. However, the datasets used are not sufficient for our task as we need a variety of flights flown by different types of UAVs to create a robust classifier. The reconstruction loss between safe and faulty states would be higher, indicating an anomaly. The two datasets used were the ALFA [11] and the UAV attack [12] datasets consisting of 47 flights and a single flight, respectively. The ALFA dataset was from flights running ArduPilot whereas the UAV Attack dataset used PX4. They generated additional samples by splitting the single flight into multiple chunks. Chowdhury et al.’s [13] approach to anomaly detection involves an autoencoder on IMU data using Rosbag files. They used the files from UAVs that ran ArduPilot and collected compressed sensor data and images at respective time frames. Their autoencoder portion is quite similar to Park et al.’s approach mentioned above. Since our focus is more on flight logs and the data captured during each flight rather than images, much of their work is not applicable.

Lang et al. [9] used an LSTM to determine whether a flight is faulty by checking if the prediction and the correct value’s residuals are within an uncertainty range. Their LSTM approach better incorporates the temporal element of flight data, especially due to their use of a window length optimization technique that leverages the recent past to help with future predictions. Our approach obviates the need for sliding windows by using the complete flights, Additionally, the structure of Lang et al.’s network changes throughout training because they only take one sample at a time; we preprocess our data to be trained in batches, maintaining a consistent structure in the LSTM.

An alternate approach to simplifying time series flight logs was proposed by Bronz et al. [10], who concatenated multiple time steps as input to a support vector machine (SVM). For example, when concatenating eight features and 20-time steps for each feature, a sequence of length 160 is generated. The data from their experiments are extracted from flights using the Paparazzi Autopilot System (PAS), but is not relevant is our case since there is not enough data and variety in the types of drones. Rather than the typical one-class support vector machine (SVM) used for anomaly detection, Rahman et al. [14] leverage a two-class SVM. They utilized sensor measurements from PX4 flights—e.g. magnetometers, accelerometers, barometers, gyroscopes, and GPS—in addition to the duty cycle of each motor.

Given the dearth of clean data with coded anomalies, many papers on UAV anomaly detection rely on simulations. For example, Galvan et al. [15] simulated seven flight hours using PX4, Gazebo, and QGround-Control. They then added simulated anomalies and detected them using neural networks.

Most of these papers focus on anomaly detection models and do not use data containing a significant number of flights flown by different types of UAVs. These approaches also temporally aligned features to avoid needing timestamp sampling techniques. In contrast, our work utilizes the full PX4 Flight Review dataset, which required more in-depth sampling and feature selection procedures to handle all the raw data.

Machine learning using maritime data is highly relevant for our discussion, as both UAV and ship data incorporate a vehicle position that changes over time. Research in the maritime domain is more mature and there is readily-available (and mostly correct) data from the Automatic Identification System (AIS) [16].

A survey paper by Sidibé and Shu [16] provided us with an overview of anomaly detection with maritime vessels. Lei [17] first simplifies the path of a vessel using grids and identifies different types of trajectories. An anomaly can be detected by measuring the degree of suspicion for each trajectory and using thresholds.

Radon et al. [18] partitioned given vessel tracks into different segments. A clustering algorithm using comparison metrics such as dynamic time warping is applied to create clusters of similar segments. An anomaly is detected if a segment does not fall into one of these groups based on a threshold.

Similarly, Laxhammar and Falkman [19] used the latitude and longitude position of vessels and applied local outlier factors to find anomalies in sub-trajectories or a portion of the complete path taken by a vessel. They also included a more general conformal anomaly detector that is capable of detecting false alarms fairly well and improved the computational efficiency of their previous approach [20]. Jiang et al. [21] used behavior modeling and trajectory prediction using classical techniques involving Kalman filters to predict the illegal behavior of UAVs and leveraged the trusted authentication of the pilots to assist in their detection. They focus more on having a UAV security platform for real-time detection and monitoring rather than creating a predictive model using UAV flights.

Most of the research on machine learning with maritime data deals with anomaly detection and trajectory prediction. For this purpose, longitudinal and latitudinal positions are usually the only features used due to the limited movement capabilities of vessels. The source and destination of the maritime vessel are also usually known ahead of time, which simplifies representation of movement and aids in detecting maritime vessel anomalies. But it is not sufficient for detecting different types of UAVs based on the obvious differences in their mechanical capabilities.

To help us refine our goal of predicting UAV types, we explored existing literature on time series classification in general. When dealing with time series data, there are both univariate and multivariate forms with different algorithms for each. Univariate time series classification is much more common since classical machine learning models and neural networks can be used. Yan and Oates [22] used multilayer perceptrons (MLPs), fully convolutional neural networks (CNNs), and ResNet on 44-time series datasets from the UCR data repository, which includes domains such as medicine and biology. For multivariate time series classification, Seto et al. [23] applied a modified dynamic time warping algorithm to classify human activities. Zheng et al. [24] used multi-channel deep CNNs for multivariate time series classification. They acknowledged that the UCR data repository mostly contained univariate time series data and is comprised of primarily small datasets, which do not work well with CNNs. Instead of looking for a different dataset, they combine two single datasets of left and right thorax heartbeats. Since multivariate time series data can be treated like 2D images, CNNs are viable. CNNs may be useful with sequential data when sequences are quite long. They are often used to avoid exploding gradient issues during the backprogagation process of recurrent neural networks (RNNs).

The closest work that aims at UAV classification using temporal sequences is research that involves the detection of UAVs based on spectrograms extracted from radar, radio frequencies, or acoustic fingerprints. Molchanov et al. [25] used micro-doppler signatures from radars to classify 11 different types of flying objects, including fixed-wing UAVs, helicopters, and quadrotors. Classical ML techniques such as support vector machines and naive bayes were employed to classify the different classes. Mendis et al. [26] extracted Fourier transforms from microdoppler signatures and used deep belief networks to predict from three different types of flying objects (artificial birds, helicopters, and quadrotors). However, there were only 70 instances of each class, not including 50 additional data-augmented instances. Brooks et al. [27] extracted 2D points from radar, fed them into wave equations, and returned a temporal series of points. These series were predicted using CNN, RNN, and MLP models. Although each of these approaches achieved greater than 85% respective to their evaluation methods, only a small subset of flight characteristics was used for classification. Location is primarily used and represented through radar or radio frequencies which do not fully capture the variations of flights. Additionally, there has been research in identifying and detecting UAVs from the audio produced by their motion. Al-Emadi et al. [28] applied CNNs, RNNs, and convolutional recurrent neural networks (CRNNs) to identify UAVs based on their acoustic fingerprints. Similarly, Jeon et al. [29] used CNNS, RNNs, and Gaussian Mixture Models (GMMs) on the same task. They simulate an outdoor setting by artificially adding noise to UAV sound data then used deep learning to predict if that sample contained a UAV. The data used in these papers is more aligned with what would be relevant in an application setting. For this reason, we would want to move more in this direction. However, we do not have access to this type of data and used more readily available data that provides an ample amount of information about flights.

Our proposed approach to detecting UAV type incorporates techniques from existing literature and various modifications to handle an unexplored dataset. The main similarity between our approach and past time series classification research is that we preprocess our time series data in a way that can be classified using a many-to-one LSTM. The architecture effectiveness has been demonstrated in a variety of previous research [9, 30]. Other than this similarity, we explore more logs, attributes, and drone types in the PX4 data and devise various novel techniques along the way to accommodate the unexplored data.

First, we use PX4 data much like many of the approaches in the UAV anomaly detection papers; however, we use significantly more data available in [31]. Second, our initial approach does not involve chunking the flights into multiple flights. We also test more timestamp sampling techniques to represent a flight better rather than performing concatenations of time series data as done in [10] or randomly selecting single points from equal intervals as done in [8]. Third, we explored more feature subsets and looked beyond the more typical features such as x, y, and z local positions and roll, pitch, and yaw angles. Even though we do not exhaustively test all of the features available from PX4, we outline a procedure that can find potential feature candidates for future testing. Lastly, we explore data augmentation and class sampling techniques as attempts to tackle the class imbalance problem. Class imbalance is not an issue in the time series classification literature that we explored but is a huge problem in our setting since we intend to use as much PX4 data as possible.

Having a comprehensive dataset with various types of flights and UAVs is of incredible value in this domain. However, the issue of data cohesion arises because of these variations. Individuals have their own goals and purpose for uploading their ulog to the website. For instance, there is a significant spike of flights that are less than 30 seconds, shown in figure fig. 2, indicating these are probably testing logs to confirm everything is working properly with the vehicle. These varied individual goals may generalize poorly to the goals of the flights that we want to detect. For example, a fixed-wing UAV flown at a low level for testing purposes would drastically differ from long-range, high-altitude flights in a surveillance task. As a result, we need to create a robust model capable of learning the behavior of sensors in relation to the different usages of quadrotors, hexarotors, and fixed-wing UAVs.

Understanding the mechanical differences between quadrotors, hexarotors, and fixed-wing UAVs is crucial for selecting the appropriate features and interpreting results. Quadrotors and hexarotors are both multirotors, which can make sharp turns and move vertically. This mobility has a drawback of being very power inefficient relative to the other vehicle types and limits flight times to around 20–30 minutes. The respective use cases will likely change as the number of rotors in a multirotor increases. More motors allow for increased payload capacity and shifts the use case to transportation, whereas fewer rotors are better for inspection and aerial photography, as smaller UAVs can fit into tighter spaces. The fixed-wing UAV is designed to function similarly to an airplane, having one rigid wing to provide lift. Its rigid wing allows for a more power-efficient method of staying in the air, allowing it to cover longer distances and carry more weight. Although fixed-wing UAVs can stay aloft for 16 or longer hours, pilots cannot place them in hover mode, making them more challenging to operate. The alternative is to fly them in a pattern such as circle, which can be inconvenient.

Understanding the difference between the flight abilities of multirotor and fixed-wing UAVs gives us an intuition that a classifier could distinguish between two types of drones. Furthermore, if we look at fig. 1, we can see the topological difference between the different UAV types. Multirotor have different flight patterns depending on whether it is waypoint-based flying (Shown in purple) or manual (displayed in the rest of the colors). Multirotor in missioned-based flights make sharp turns going directly from one waypoint to the next. When flown in manual mode, the paths turn more curved as that is the more natural form of flying. On the other hand, fixed-wing UAVs, independent of the flight mode, have a slower turn, producing a more curved path, as shown in the flight’s multiple loops. These patterns might indicate that when flight mode is a mission type, the system will have an easier time telling multirotor and fixed-wing flight apart. Telling a quadrotor and hexarotor apart seems more tricky, as the flight characteristics look visually similar.

For our classification task, we downloaded every log but only kept the quadrotor, hexarotors, and fixed-wing UAVs. Other types of drones in the PX4 data include octorotors and ground rotors, which were not used because we felt that having two similar (quadrotors and hexarotors) and different (quadrotors and fixed-wing UAVs) types of drones would be sufficient. Figure 1 shows two flight examples for each vehicle’s type. We initially chose quadrotor and fixed-wing UAVs due to distinct flying characteristics as proof of concept that our method could detect the difference between these vehicle types. We then expanded the classifier to include hexarotor to see if it was possible to predict vehicle types that were considerably more similar (hexarotor and quad rotors). This extraction resulted in 29,362 flights after removing the logs that had parsing errors from the library we used. Of these flights, 26,706 logs were from quadrotors, 1,332 from hexarotor, and the remaining 1,324 were from fixed-wing UAVs. Clearly, there is a significant imbalance in the number of quadrotor flights compared to the number of hexarotor and fixed-wing flights. This imbalance can be explained by the fact that quadrotors are cheaper and have recently become popular for recreational use. The amount of data is further reduced in each class after feature selection, see section IV-B. This imbalance may lead to a challenging classification task if the appropriate methods are not used.

We can better understand how to engineer our data to be used for classification by beginning with the model. We considered the fact that even though this data is time series, we are not performing a forecasting task. As a result, we are more interested in models that can learn the temporal nature of the data and output a single value that represents the predicted class. One possibility is using dynamic time warping (DTW), which combines Euclidean distance and dynamic programming to compare time series data, and is typically used for univariate data [23]. Even though there are modifications to this algorithm to accommodate multivariate data, this method is time inefficient with the size of our data since it compares time series data in quadratic time. Exploring DTW is not ideal since our data is not well explored and requires a significant number of training experiments.

We could use more classical ML models by manipulating the data such as SVMs, Naive Bayes, or decision trees. However, the general reason these models are not used is the high number of features. Using these models would require concatenating each feature, which neglects the time component and may also suffer from a high number of dimensions [33]. Additionally, SVMs do not support multi-class classification and are designed for binary classification. The workaround is to create multiple binary classifiers, which for our case, makes it difficult to compare the effects of a majority class’s distribution to other minority classes. It is possible to reduce these problems mentioned above by using deep learning models.

An example of a potential deep learning method would be multi-channel CNNs as performed in [24]. Multi-channel CNNs involve breaking down a multivariate task into several univariate tasks. Then, a series of feature extractors and hidden layers are used with a final MLP for classification. However, our primary objective is to see how changes to data preprocessing and tackling class imbalance affect performance instead of focusing heavily on the model’s architecture. We want our pipeline for classifying UAV types to serve as a starting point in which models can be swapped out and tested.

RNNs are used frequently in time series tasks, especially for forecasting. Since time series data is inherently sequential and order matters, the memory capability of RNNs is crucial. Therefore, we consider RNNs as an intuitive model for our data. Our pipeline uses Long-Short Term Memory networks (LSTMs), a variant of RNNs. LSTMs address the problem with vanishing gradient in RNNs that occurs much more frequently because of the number of time steps [34]. Solving this problem also leads to computational efficiency. LSTMs have the downside of not being able to capture long-range dependencies. However, our task does not require it for two reasons. Specifically, a simple LSTM with a single layer and 128 LSTM cells was used for our experiments to great effect, see figure 1 in our supplemental material (https://osf.io/mnsgk/) for model illustration and Section V for results. This single layer is paired with linear classifier output. By configuring this model to take in an input equal to the number of binned intervals, a many-to-one LSTM and classifier were created to classify if a flight sequence is from a quadrotor, hexarotor, or fixed-wing UAV. Once we have decided on a model, we can focus on preparing the data for the LSTM, starting with feature selection.

There are major challenges when handling the entire available set of raw PX4 flight logs. The data for each flight is comprised of topics, which are a collection of features derived from the same sensor [31]. For example, the topic ‘vehicle local position‘ will likely contain ‘x‘, ‘y‘, and ‘z‘ features relative to the starting position. The PX4 data set contains over 325 of these topics. However, only 27 topics are contained in at least 60% of the flight logs.

Another challenge is that we were unable to find consistency in these topics with a particular software version or hardware type. In some instances, a topic missing makes sense, such as some fixed-wing lacking battery information, as a fixed-wing can run off of gas power. On the other hand, some topics may vary in naming convention despite capturing the same data. We learned that these topics have evolved over time; therefore, finding a common set of topics to aggregate a processed data set for model training and evaluation is difficult. This characteristic of inconsistency also holds true for the features of each topic.

One possible technique for feature selection involves cycling through all permutations, such as sequential forward selection (SFS). SFS finds a potential feature subset by gradually adding a single feature [35]. At each iteration, the best-performing subset is chosen, and we continue the process with the permutations for the remaining features. This approach, along with other feature selection methods, assumes that all features are equally contained across samples. However, with our data set, that is not the case. This data inconsistency creates instances where a combination of features can really limit the data’s availability. Therefore, we have to be careful in our feature selection.

It is unrealistic to exhaustively test all combinations, but at the same time, we do not want to constrain ourselves to a very small subset of features; therefore, we tried several possible feature subsets based on varying criteria. This approach ensures that we explore more than just the features proposed in research related to our task and does not require cycling through various features that may not have any value in prediction.

The first step of this process involves only checking topics if they are contained in at least 60% of the logs. Since many topics may be unique to specific logs, we can prune these less-common topics out. Additionally, even if logs have the same topics, they may not necessarily contain the same features. Therefore, we applied the same technique of using only features contained in at least 60% of logs. These steps reduce the feature search space and allow us to maintain a reasonable dataset size for testing.

We begin with a base set of features from [8] to reduce the variability that comes from completely random features. These features are the x, y, and z local positions as well as the roll, pitch, and yaw angles. Next, we select the number of feature subsets we want, randomly select $n$ features without replacement from the pruned features, and manually remove some of the features. Manual removal requires some assumptions to be made to simplify the process. We consider features not related to motion and/or time as irrelevant. For example, within the ’vehicle gps position’ topic, there are features called ’satellites used’ and ’jamming indicator’. Both are more status-based features that can be manually removed.

Given these feature subsets, we can perform experiments with our model to give us a rough estimate of which features work best. Details regarding the number of feature subsets tested and the features used are clarified in the experiments section (section VI).

We acknowledge that this approach is not exhaustive, and that the quantity of data will change with each subset. As a result, we recognize that the variations in performance using these feature subsets may be a result of the different derived class distributions.

Before we can determine an approach to handle the variable durations, we first need to understand that this process is further complicated by the fact that there are multiple features per sample that will likely sample at different times and at different rates. The most straightforward way of addressing this issue it to align these features temporally and apply zero padding to the beginning and end of shorter sequences. One downside to this approach is that the data may become very sparse since shorter sequences may become mostly zeros if much longer sequences exist. This problem is not as severe in our task considering that we have a much smaller subset of features that are similar in beginning and end timestamps. For example, the angular acceleration and local position sensor values are likely to be aligned because of their similarity. Our approaches are based on zero-padding with modifications to determine which values are sampled from the aligned features. Once we assign a way to align the features, we can proceed with sampling timestamps to bin and summarize the data.

Timestamp binning refers to the sampling method used to convert the time series instances from variable to fixed length. The simple approach is take the maximum and minimum of timestamp of each log and divide the timestamps into equally sized intervals called equal-width binning. We produce a dataset of equal length time series data because the number of intervals are fixed. This local maximum and minimum timestamp per log ensures that each feature sampling lines up at the interval timestamps since there are multiple features per sample,. Having a higher number of intervals increases the resolution of each part of the flight. However, the differences in durations make it difficult to find an appropriate number of intervals to accommodate these variations, which we will further discuss in our experiments. We reduce the number of outliers that could skew the range of timestamps by performing this independently for each log [36]. Once we have these intervals, we need to obtain a single value to represent the values within each interval.

There are drawbacks and benefits to the different approaches to obtaining a single value. Park et al. assumed that a randomly selected value could represent a feature during a time window [8]. In their setting, this approach makes more sense because of the short duration of each of their instances (30 seconds). However, because of the longer durations of our flights and the variations, taking a random sample would not accurately represent a full interval. Taking the maximum or the minimum within each interval suffers with sensitivity to outliers and may not adequately represents parts of a flight log with a significant amount of movement and change in directions. We intend to sample data in a way that captures more information about the flight or aligns data between different flights.

Here we detail the two sampling approaches we compared to determine how to engineer the data to fit our model. The first approach we refer to as average sampling, which divides each flight into an equal number of bins based on a local minimum and maximum timestamp for each flight. The second approach we call fixed window average sampling, which essentially shrinks the number of timestamps used in the first approach to a specified time range.

We perform some experiments prior to comparing timestamp sampling and class imbalance techniques. These experiments help establish a starting point for our compared techniques.

After these preliminary experiments, we define the components of the pipeline that are fixed. The preliminary experiments gave us the following features: the x, y, and z local positions, the roll, pitch, and yaw body angles, the throttle, altitude, and battery temperature. They also gave us the training and model hyperparameters such as a 0.001 learning rate, Adam optimizer, and a single hidden layer LSTM with 128 nodes. We can then proceed with these features and hyperparameters.

The first experiment involves trying different timestamp sampling techniques and analyzing their effects on model performance. We begin with timestamp sampling since choosing a technique does not require any changes to the distribution of classes. Fixing variables and choosing this experiment order still effectively shows the benefits and drawbacks of each tested technique. Furthermore, this approach reduces the variability that comes with changing many variables at once. We test a set of hyperparameters for each technique in the following experiments.

The results of our two main experiments are shown in Tables II and III. For Experiment 1, average sampling using 50 intervals led to the highest performance. After averaging across 10 folds, we reach f-scores of 98.16%, 73.15%, and 42.15% for the quadrotor, fixed-wing, and hexarotor classes, respectively. The overall performance of this technique can be summarized with the macro f-score, which was 71.15%.

We proceeded with this sampling method and number of intervals for Experiment 2, in which performance did not improve across all of our attempts to address class imbalance. If we look at the results of Experiment 2 independently of the first experiment, we achieved the best performance using data augmentation with a 200% increase to the minority class. This configuration led to f-scores of 96.09%, 74.67%, and 37.8%, for the quadrotor, fixed-wing, and hexarotor classes, respectively.

There is a challenge of qualifying these results since there is no baseline of comparison, but we can think from the perspective of choosing only the quadrotor class since it is the majority or randomly guessing the UAV type. This approach would only give a macro f-score of 32% since all the minority classes are not predicted. By guessing each class with a 1/3 probability, we on average reach a macro f-score of 20%. Guessing each class based on the distribution of classes would yield similar results to guessing quadrotor each time.

The analysis performed on these results is done both quantitatively and qualitatively. While we can simply look at the resulting performance of each modification, there are other factors to consider, such as speed and ease of implementation. These factors are important when using such models in real-life settings, especially if performance is relatively consistent across techniques. We can make more accurate comparisons of different techniques by analyzing the averages across ten folds of data.