A Concept on the Shared Use of Unmanned Assets by Multiple Users in a Manned-Unmanned-Teaming Application

1.1 Manned-Unmanned-Teaming

Manned-unmanned-teaming (MUM-T) describes the process of teaming the crew of a manned command vehicle with one or more unmanned vehicles in order to increase mission efficiency and effectiveness [1, 2]. The unmanned assets can support with capabilities such as reconnaissance, surveillance, engagement, or electronic warfare to achieve a given mission goal.

The overall idea with MUM-T is to combine manned and unmanned vehicles to increase mission efficiency by merging their advantages and compensating each other for their respective disadvantages [2]. In these teams, humans contribute most significantly with their cognitive abilities, such as problem solving, mission planning, the recognition of interconnections and their individual experience [3]. These qualities as well as the authority to decide about potential weapon use make the human vital in such a team. Unmanned vehicles on the other hand, can contribute to mission success with support in reconnaissance by automated sensing and perception, the increase of the sensor range of the manned system and the automated task execution. In particular, remaining distant to potential enemies or potentially hazardous areas can reduce the risk for the manned command vehicle.

1.2 Multiple Users in a Manned-Unmanned-Teaming Application

Previous work studied concepts of controlling multiple unmanned vehicles in manned-unmanned-teams [1, 4,5,6]. However, these studies imply that the unmanned vehicles are at one user’s exclusive command. Due to their utility however, it is plausible that additional users will have need in using these assets. Such a secondary user might be acting cooperatively with the primary user on the joint mission, or be acting independently requesting the unmanned vehicles for their unique purposes.

Besides the advantages of making limited resources available to other users in a MUM-T application, we also expect disadvantageous effects on the primary user’s workflow and situational awareness (SA). The user’s mental picture of availability, location and planning of resources can be adversely affected by uncoordinated foreign access. Such SA issues are most likely associated with an additional loss of performance, especially when resource capacity is insufficient to cover the additional demand. We aim to reduce SA and performance issues and increase the effective use of the unmanned vehicles by supporting a coordinated use between the users.

In this section, we describe our concept of how multiple users can be given coordinated access to a limited number of unmanned assets. For simplification, we initially consider only two users and a single unmanned vehicle. However, the concept can be applied to multiple unmanned vehicles and multiple users. To develop this concept, we have oriented ourselves on applications of third-party services, such as artillery [7], medical evacuation [8], search and rescue [9], close air support [10], and unmanned aircraft system handover [11].

2.1 Distribution of Roles

When different users access and control an unmanned vehicle, it is crucial that there is a defined role distribution including continuous accountability for the asset. To this end, we define different roles with associated rights that are assigned to the individual users. Derived from already established definitions in computer science, we will use the terms “host” and “client.”

The host is responsible for an unmanned asset, makes it available to other users and coordinates its use. The host can either benefit from the asset or only take on a coordinating role. The client represents the external user who wants to use the capabilities of the host’s asset. To this end, a request can be directed to the host that specifies the desired use. Depending on the urgency, capabilities and availability of the resources, the host has the ability to accept, reject, modify or offer an alternative to the request. Depending on the requirements, the host can grant the client access to the unmanned asset. However, the host will always be responsible for the asset and also has the necessary authority to revoke the client’s access when and if desired. Whereas only one user can be host of an unmanned asset, multiple users can request its use as clients.

2.2 Levels of Shared Use

To meet the variable requirements of both the client and the host, we define multiple levels of resource sharing depicted in Fig. 1. The access by the client ranges over different levels from a purely passive use over an indirect use by means of a request, up to the exclusive active use. Depending on the respective level, the use of the asset by the host is reduced accordingly. It is thereby presupposed that the client holds the necessary competence to access and control the asset.

Levels of shared use.

Full size image

Level 0.

This level serves as a reference as the client is not granted any use of the unmanned vehicle. The host has exclusive usage rights over the asset whereas the client cannot take any advantage of it.

Level 1.

On this level, the client is granted passive use of the asset. The client receives results emerging from the host’s current use, but cannot actively influence the generation of results. Although the client has no means of using the unmanned vehicle based on own demands, this level offers the opportunity to share the sensor outcomes of an unmanned vehicle without interfering with its control. Thus, this level can support a shared situational picture of the environment.

Level 2.

This level enables the client to make unmanned vehicles related task requests to the host and thereby grants indirect control of the asset. The host is required to implement the task and can therefore decide whether to execute it as requested, modify, or reject it. Although this form of shared use increases the coordination effort for the host, it allows them to adapt the requests to their own needs and intentions.

Level 3.

While the host was actively using the resource on the previous levels and had sole direct control over it, control is relinquished to the client on level 3. Accordingly, the client is no longer subject to the host’s approval of tasks as on level 2. Once the host grants active use, the client can control the asset at their own discretion. However, the host still has the responsibility over the unmanned asset and the authority to revoke the client’s usage rights. Consequently, the client can use the resource widely without restriction at this level, yet the host can regain full control of the asset at any time, e.g. because of own demand or due to safety reasons. This level may require less coordination effort for the host because only the unmanned vehicle and the time period have to be assigned to the request, however the corresponding asset is then temporarily unavailable to the host.

Handover.

Levels 1 to 3 represent different possibilities for the shared use of an unmanned vehicle, with control continuously shifting from host to client as levels increase. However, the responsibility and authority over the resource remains unaffected and lies always with the host. The remaining option is to transfer responsibility and authority from the host to the client in a handover. This process changes the role distribution in that the client becomes the new host of the asset. The former host becomes a client who is required to make a request to use the asset again.

2.3 Strategies for the Distribution of Resources

Based on the levels 2 and 3 of shared use, various strategies result for structuring a network of several users and several unmanned vehicles. The examples (a)–(c) in Fig. 2 illustrate the distribution of several assets between two users, with one of them hosting all assets. Example (d) presents a configuration with multiple clients and distributed responsibility.

Strategies for the distribution of unmanned vehicles

Full size image

Distributed Shared Use.

The unmanned vehicles in example (a) are distributed in such a way that each is used exclusively by one user. Accordingly, the client is either not allowed to have any use (level 0) or shared active use (level 3). Since they are each used exclusively, coordinating their use between multiple users is not required. However, the allocation of the unmanned vehicles to the respective user must be coordinated. Apart from a change in the available resources, host and client are not influenced by each other. It is not necessary that the host has control over at least one unmanned vehicle. It is conceivable that all unmanned vehicles are used exclusively by a client.

Mixed Shared Use.

In example (b), some of the unmanned vehicles are used exclusively and some are shared to be used by task-requests. The usage of shared assets must therefore be coordinated so that host and client can make optimal use of them. At the same time, however, assets can be reserved for exclusive use so that conflicts arising from shared use can be cushioned. In addition to the allocation of the users to the unmanned vehicles, the coordination effort is increased accordingly by the coordination of shared use.

Collective Shared Use.

In the configuration of (c), none of the unmanned vehicles are used exclusively by a single user. Instead, all assets are shared to be used by client task requests. This requires a very high coordination of their use, yet it provides the possibility to make maximum use of free capacities.

Multiple Clients and Distributed Responsibilities.

Until now, we described the distribution of unmanned vehicles between two users, where one acted as host and the other as client. However, responsibility for multiple assets as well as their use is not limited to two parties. Example (d) shows a configuration with three users in which responsibility for the unmanned vehicles is distributed among two of the users, each of whom acts as host, and one acts purely as a client. There is a distributed shared use between host A and client A. Host A, however, is also a client to host B and accordingly supporting user and supported user (client B) at the same time. Host B does not use the own asset, but has a purely providing function. However, the asset is not available to a single client, but is shared by two clients.

Our specific research focuses on the teaming of a manned two-seated helicopter and unmanned aerial vehicles (UAVs) to accomplish dynamic and complex military missions [12]. In our scenario, the helicopter crew has multiple UAVs at their command that support the reconnaissance of mission-relevant areas and serve to increase the helicopter’s sensing range. They are controlled according to a task-based guidance approach [13, 14]. This approach provides that the UAVs do not have to be controlled manually, but can carry out high-level mission tasks given by the helicopter crew. Using automation on board the UAV, this complex task is broken down into executable sub-tasks that can be processed in sequence.

In addition to the helicopter crew, a further user is integrated into the mission scenario, who also requires the capabilities of the UAVs to accomplish their own mission. This role will initially be assigned to a ground-based unit such as a forward air controller or a convoy leader. However, this secondary user is not limited to be ground-based, but could also be an airborne unit like another helicopter or a jet fighter.

3.1 Identified Role Distribution

Within our application, we define that the helicopter crew is responsible for the UAVs and therefore acts as host. The additional ground unit serves as client and is authorized to make requests for the temporary use of the UAV services. As host, the helicopter crew is required to decide whether to accept, modify, reject or cancel requests made by the client.

3.2 Identified Levels of Shared Use

In order to interfere as little as possible with the helicopter mission and pilot’s use of the UAVs, the scope of usage by the client should be flexibly oriented to its minimum requirements. By applying the levels of shared use to our application, we identified three types of client requests. The transfer of authority is not considered in the following because it is currently not intended to be applied to our application.

Level 1.

The client is permitted to receive the UAV’s live sensor data and the results from its automated sensor perception. However, no control over the sensor or the UAV is granted. The client cannot actively manipulate the generation of results and is therefore merely a passive recipient. Since only the sensory output is shared with the client, there is no interference with the pilots’ use of the UAVs or their mission plan.

Level 2.

By requesting specific reconnaissance tasks, the client is enabled to influence the tasking of the UAVs. The requested tasks have to be integrated by the helicopter crew into their own mission plan by assigning them to the schedule of a specific UAV and are therefore subject to their approval. Modifications or rejections of requests have to be tolerated by the client.

Level 3.

The client is temporarily given control of the UAV and can thus use it independently to accomplish the individual goal. During this period, the helicopter crew cannot use the UAV for their own needs. However, the responsibility and authority over the UAV remains with the helicopter crew, who can withdraw control from the client at any time.

3.3 Requirements for the Effective Shared Use of Unmanned Vehicles

To enable the effective shared use of unmanned vehicles, it is necessary to identify the requirements for each user. However, we will not discuss technological aspects, such as means of data transmission, but functional aspects of the interaction between host and client. In this context, it is important to consider the relationship between host and client and their specific requirements, depending on whether both users act in a common mission space, require similar situation information or pursue a common mission goal.

Networked Distribution of Tactical Information.

According to our defined levels of shared use, the exchange of tactical information concerns only such data that was generated by the respective shared unmanned vehicle. However, tactical information with an origin other than the asset can be essential for an effective sharing of unmanned vehicles. The demand for third-party tactical information is bidirectional. For example, the host requires information from the client to verify how the unmanned vehicle is used in order to wield authority. Additionally, information distribution between cooperating units that operate in a common battlefield is important for overall information superiority and efficient cooperation. After all, it seems contradictory to provide support through the exchange of resources while withholding tactical information in this matter.

However, information distribution between two users is highly dependent on their degree of cooperation. Exchange may include general tactical information (such as positions and attitudes of forces or other kinds of threats), vehicle specific tactical information (such as airspaces, flight corridors or no-fly zones for airborne systems), or mission specific tactical information (such as flight or march routes, mission objectives, points of interest or time over targets).

Request of Resources.

Requesting the use of an unmanned vehicle requires information about the availability of the asset and a specified form of the request. A well-defined form of requests can be found in many applications of operational support, such as the artillery, medical evacuation, search and rescue, or close air support [7,8,9,10].

Information about the availability of resources and the associated host is crucial to enable the client to make a directed request. The extent to which information on resource availability is shared is again subject to the cooperation between client and host. Information distribution can range from a simple notification that an unspecified number of assets is available to the provision of asset-specific information such as number, capabilities, and scheduled tasks.

A well-defined form and content of the request should ensure that the request is complete. Moreover, uniformity and conciseness should contribute to quick perception and comprehension by the host. Thereby, we have to differentiate between mandatory and optional parameters. Mandatory parameters define the minimal content of a request and thus guarantee that the request is complete. They must also always be defined by the client and should not be modified by the host. If only the mandatory parameters are defined, the host has the greatest possible flexibility to implement the request. Optionally defined parameters allow the client to specify the request more precisely, but in return limit the host’s flexibility in implementing the request. The client should therefore be able to indicate whether an optional parameter represents a condition or a preference.

Mandatory parameters are the requested level of shared use, a priority, and, if applicable, the requested UAV task in a level 2 request or the duration of use in level 1 and level 3 requests. If an explicit task is requested, the duration is automatically derived from the duration of the task. Optional parameters are, for example, a specific start time or a specific UAV.

The priority describes the urgency of the request and its importance for the client’s mission. The choice of priority should aim to minimize interference with the host’s mission plan. For our application, we have defined the three following priorities:

• High: The request is time- and mission-critical (e.g. due to a threat to the client).

• Medium: The request is mission-critical but not time-critical.

• Low: The request is not mission critical, but contributes to mission efficiency.

Provision of Resources.

To provide the service of an unmanned vehicle, the host has to verify the request for its general feasibility and then assign a start time and a UAV if these were not specified in the request. Thereby, the host should orient on the priority and the specifications of the client and check whether parameters need to be changed or defined. Since the helicopter crew has to integrate the client requests into their own mission plan, it may happen that they can only approve requests under adjustments. In general, these should be made manually. Alternatively, it should be possible to delegate the modification of these parameters back to the client by indicating the parameters that need to be changed. This should help to reduce the task load of the host and allows the client to choose an alternative at own discretion.

We presume that the workload of the helicopter crew is increased in MUM-T applications, because in addition to controlling the own helicopter they are also responsible for the management of the UAVs [2]. Therefore, our institute has been working on different approaches to support the crew by an assistant system. One approach pursues the support in mission planning and re-planning by applying a mixed-initiative approach. More information on our application can be found in [15, 16].

We expect that the manual processing of the requests will result in an increased task load and workload of the helicopter crew. Therefore, we aim to extend the support by the assistant system to the processing of requests. Assistance shall be provided by verifying the general feasibility of a request and by suggesting a suitable implementation.

4.1 Pre-processing

In order to generate effective responses to requests, the assistant system needs to access additional knowledge from a mission planner. This planner has already been developed to assist in the field of mission planning and re-planning and has knowledge of the current mission plan, including the planned tasks for helicopters and UAVs [15, 17]. Since the mission planner also has knowledge of the mission objective, it can identify tasks that have not yet been planned by the helicopter crew but are important for the completion of the mission. With this knowledge, the assistant system should be able to check the general feasibility of a request and generate a suitable implementation, i.e. to determine which UAV to assign and the start time for an associated request. The basis for this is an optimization considering both the current mission plan and the parameters defined by the client.

4.2 Post-processing

Within the pre-processing, the assistant system determines the best solution in terms of its optimization metrics for implementing the request, although other solutions may also exist. In this way, the host will be supported with the automated generation of alternatives to the initial solution proposal in a post-processing. The assistant system should provide a selection of solutions if several possibilities for implementing a request are identified.

Alternatives can be generated by the assistant system either based on constraints imposed by the host or by free variation. In both cases, this will result in either the assigned UAV, the start time or both being changed. The assistant system then generates a new optimized solution proposal under consideration of the additional constraints. Thus, the generated alternatives may be worse than the initial solution proposal in terms of the assistant system’s optimization metrics or they may violate the specifications selected by the client. However, for reasons that are not available to either the client or the assistant system, such as a tactical preference by the pilot crew, it may be necessary to select an alternative.

4.3 Integration

Whereas the automation on board the UAVs is controlled in a hierarchical manner, the assistant system is intended to be in a cooperative relationship with the users. In order to support the processing of client requests, the assistant system should serve as a central gateway for all communication and interaction between host and client. Under this premise, requests can be pre-processed by the assistant system before the pilot is confronted with them. Thus, a generated solution proposal can be presented simultaneously to the corresponding request. Figure 3 shows our application in work-system notation [18]. Visible are the two work processes of the client and the host. They are linked through the interactions of the request and the provision of UAV services. Within a work process, a distinction is made between the workers and the tools. A worker knows and pursues the work objective by own initiative. To achieve it, the worker uses tools in a hierarchically degraded manner. Depending on the automation, it can be located either on the side of the tools or on the side of the workers. The process of mission planning takes place by hierarchically using the planning tools for helicopters and UAVs (green arrows) and can be performed actively by the pilot or passively by the assistant system. Pilot and assistant system are linked cooperatively through the interaction to implementation proposals (blue line).

Our MUM-T application in work-system notation (Color figure online)

Full size image

In order to investigate automation and pilot assistance in the scope of MUM-T applications, our institute operates a helicopter research simulator. A generic helicopter cockpit serves to simulate the conduct of military transport missions in combination with three UAVs. The workstation of the ground unit is simulated using a handheld device, which can be used to formulate UAV access requests.

Based on our concept and the application-specific considerations, we have developed a prototype of an assistant system and integrated it into our research simulator. Furthermore, we have implemented interfaces for host and client and the interaction between them and the assistant system. These interfaces are, however, prototypical and subject to continuous development.

In the following sections, we will illustrate the interaction between a helicopter crew and a convoy operating in a common mission area. Thereby, we discuss the request of a reconnaissance task by the convoy and the subsequent interaction between the assistant system and the helicopter crew.

5.1 Interface Client

With the handheld device, the client has access to an interactive tactical map of the simulated operation area with all provided tactical situation information. This map also provides information about the host and available UAVs and serves to initiate and specify UAV access requests.

The requests initially contain the priority, the requested level and, if applicable, the specific task. Using an adjustment menu, the client can then specify the requests by various factors, such as the duration or an explicit start time of the requested service. In case that the appropriate information about available UAVs (especially their capabilities, positions, and tasks) is provided, the client can also specify the particular UAV.

Figure 4 shows aspects of the client interface. To the left, the tactical symbol of the own unit is shown (blue circle) as well as the planned route (black line). By selecting the upper section of the route, the depicted context menu opens and offers different actions for the referenced object (green bordered). Among other actions, a level 2 request for a recon task can be made here. The right side shows the specification menu after selecting the recon request. The request and the associated reference object are already defined by the general context menu. The priority is preset, but can be adjusted. The UAV and the start time can be selected as optional parameters. After confirmation, the request is forwarded to the host.

Parts of the client interface (left: context menu, right: specification menu) (Color figure online)

Full size image

5.2 Interface Host

The helicopter crew is provided with an interactive map in their displays, which is similar to the client’s map but with an extended functionality. It serves, among other things, for overviewing the tactical situation, for tasking the UAVs and for processing client requests. Figure 5 shows the presentation of the request and the corresponding solution proposal that the assistant system has determined during the pre-processing phase. A dialogue box is used to present the request in text form. The yellow exclamation mark indicates that the request has a medium priority.

An illustration of the host interface (top: dialog box, middle: visualization of proposed task) (Color figure online)

Full size image

Below this, also in textual form, is a summary of the proposed solution, which proposes the immediate assignment to UAV1 (orange). In addition, the proposed solution is shown on the tactical map (in the middle of the illustration). Embedded in the mission’s situation picture, the proposed solution is displayed as a blue-framed mission arrow, which connects the selected UAV with the location of the requested mission. Using the dialog box, the host can decide whether to accept the proposed solution, decline the request, or have an alternative generated by initiating the assistant system’s post-processing.

We described our concept for sharing unmanned vehicles in a MUM-T application, which aims at allowing multiple users coordinated access to unmanned vehicles. To this end, we defined two different roles with associated rights and responsibilities. The host is the supporting user who provides access to an unmanned vehicle. The client is the supported user who can request respective services. To enable a variable provision of the unmanned vehicles, we defined three different levels of shared use. These levels allow to flexibly transfer control from host to client. We then applied this concept to our application and identified further specific requirements for requesting and providing unmanned vehicles. We furthermore described an assistant system to support the host’s decision-making by proposing suitable implementations of requests. Finally, we presented our approaches to the implementation in a helicopter research simulator, in particular the corresponding human-machine interfaces. As an example, we demonstrated and described the process of filing a task request and the associated interaction between the assistant system and the host.

In our application, only a single request was handled at a time. Future efforts aim to enable the processing of several simultaneous requests from one or multiple clients. This is associated with additional requirements for the request and provision of the unmanned vehicles, such as the definition of time dependencies between multiple requests from a single client. Enhancements to the assistant system include the development of different methods to implement requests, adaptive intervention behavior and variable levels of support, based on the crew’s workload.