Abstract
High-end industrial vehicle simulators are generally expensive and aim at providing a high level of realism. The access to such simulators is often a limited resource to researchers and developers who find themselves using a PC-based simulator instead. We challenge this approach by introducing a low-cost mixed reality simulator for industrial vehicles that allows to test new vehicle control concepts and design ideas in a rapid prototyping manner. Our simulator prototype consists of a head-mounted projection display, a CAVE-like room covered with a retro-reflective cloth and a rotatable chair with controls to steer an industrial vehicle. The created digital environment represents an obstacle course for an excavator and can be controlled by a joystick, a keyboard and can be explored by natural head movements.
Performed user tests with 21 participants showed that the mixed reality simulator is perceived as more realistic, natural to use and provides a more immersive experience than a PC-based simulator with the same environment and controls.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
Simulators, providing a virtual environment to users are nowadays available in different varieties and range from low-cost gaming simulators to highly realistic vehicle or industry simulators, such as professional car or flight simulators. Many development teams or researcher do not have access to high-end simulators. This might not be needed or cannot be afforded in many cases. Nonetheless, we argue that only using simple PC-based simulators might limit options for research, design and development teams when testing new concepts and ideas for industrial vehicles.
Generally, simulators provide advantages in terms of reproducibility, standardization and controllability of scenarios and tests. A simulator replicates scenarios that might be difficult, expensive or even risky to reproduce in reality, such as a dangerous driving scenarios, where a driver would be physically at risk. Furthermore, it allows controlled simulation environments that are not affected by wind, weather and other external circumstances [7]. Additionally, the use of simulators and interactions with them can be evaluated without placing humans or physical objects in a real environment or before the real environment is ready, for example when evaluating improved operator environments.
These facts would argue for using a simulator in test and design scenarios. However, despite the progress in hardware and software used for simulators, there are still limitations. Industrial simulators are usually built with computer monitors or projection solutions that can be space consuming, require long setup times and pose high costs. Therefore, initial experiments, user evaluations or even trainings are often performed on simpler simulators. Fully featured simulation environments are used in later stages of a project, when previous stages indicated positive results. This limitation and way of working results in increased costs as well as it limits early evaluation of interaction with the technology or interface.
Another alternative when making the simulation is to use virtual reality (VR) glasses, which provide the ability to create an immersive virtual reality. Simulating a virtual environment in which the user can naturally look around by moving the head to interact with the virtual environment can be important in many scenarios, such as the simulation of complex industrial vehicles (cranes, excavators, fork lifts, etc.) or even normal driving scenarios (e.g. overtaking, parking, etc.) [13, 16]. A mayor limitation with virtual reality is when physical artefacts like sliders, knobs and interior design of a real world prototype have to be used while performing in the virtual environment, e.g. mixed reality.
This paper introduces a low-cost mixed reality simulator that challenges many of the above-mentioned constraints. Our solution is easy to install, low-cost, consists of off-the-self hardware and a virtual environment that was created with the Unity game engine. In our prototype, we create a mixed reality environment while using physical controls and a projected virtual scenery. Our technology can be used in small setups where virtual windows or screens are in front of a user or to create larger CAVE-like simulations.
Furthermore, we evaluated if our mixed reality simulator provides the basic ability to be used in a scenario such as for industrial vehicles. Therefore, we created an industrial vehicle simulator resembling the physical controls and virtual representation of an excavator on a virtual obstacle course. In such environments it is of higher importance, compared to ordinary on-road cars, to be able to look up and down and to see the environment around the vehicle. In our simulator, users can explore the virtual environment freely by naturally moving their head in a CAVE-like setup.
The contribution of this paper is a low cost mixed reality simulator that can be used for testing and designing human machine interface (HMI) applications, such as evaluating new concepts to interact with vehicles, for education or for user training with industrial vehicles.
2 Related Work
Simulators have been used in many vehicle research areas to evaluate user behavior and HMIs as well as for education. These can range from specific replica simulators costing hundreds of thousands euros [24] to low cost simulators [3] and desk simulators [23]. The development of 3D engines with physics simulation and collision detection, as well as easy to use editors, also make it possible to build custom simulators [6]. For many purposes a more light weight simulator is sufficient, “as our ability to fill in the gaps to create strong cognitive representations has clear potential as an alternative to modeling every last detail of the space” [28]. In an on-road driving simulation, for example, it may be enough to look at a PC monitor to display the simulated environment in front of a car and to use a gaming steering wheel as input device. However this approach offers a very limited field-of-view (FOV), limiting simulations where operators need to move their head or body. This is especially a problem when interactions need to be tested that require to see the surrounding environment.
The interest to build virtual and mixed reality display systems [5, 25, 26] has increased, as new products are introduced to the market. Most of the see-through products offer a near-eye solution with a limited field of view, a constant focus, single eye usage (no stereoscopy), and limited depth of field. Many near-eye solutions come with a great deal of optical complexity in design, e.g., Google Glass [22], or Microsoft‘s HoloLens [19]. In [10] an additional specially made contact lens needs to be used to see the content. Thus, the users are having the challenge of interacting naturally with the physical world due to these optical limitations.
By disconnecting the user from the real world, the mentioned limitations are avoided. This is achieved via opaque wearable stereoscopic head-worn displays, e.g., Oculus Rift [8], also known as virtual reality glasses. Nonetheless, the challenge for real-life use cases remains the same or is even higher, as real world content now must be replicated into the virtual world [18].
While simulators are a great tool for assessing different scenarios and measuring driving performances, participants often face the problem of suffering from motion sickness [7]. This observation has been made in various research areas, including vehicle simulators [13] and flight simulators [17]. Motion sickness appears in mixed-reality and VR simulations respectively. Research showed, that the time spent in a simulator affects the likelihood of simulator sickness and that older participants have a greater likelihood of simulator sickness than younger participants [4]. However, participants react differently to simulation environments. Therefore, we found it important to get an indication on how our simulator behaves in this respect.
The use of laser scanning pico-projectors offers an interesting alternative, as these do not require any optical components to focus on the projection surface and the amount of pixels displayed stays constant with the increasing distance between the projector and the screen. Additionally, it comes with a coin size light engine [9] with further possibilities for miniaturization, thus making it even more wearable. The image qualities as well as the use of reflective material has been investigated before [2, 12]. Image projections can also be made onto non-reflective surfaces [20, 21]. However, in our setup we use a reflective material as this enables to mix physical controls with the virtual scenery without distortion and in different lighting conditions.
Wearable laser projectors, as we use it in our research, were used by Harrison et al. before [11]. They used a shoulder mounted projector combined with a depth camera to project images to non-reflective surfaces and to allow gestural interactions. In our approach we have further developed the idea of using a laser projector as a head-mounted mixed-reality device using a retro-reflective surface as screen. This technology was used before for motion capture acting support applications and for gaming tests [1, 15]. Unlike other systems like e.g. from CastAR [14] or other research [27] this system does not require multiple projectors. Thus, problems originated from using multiple projectors such as the keystone or image registration effect are not an issue in our system.
3 Simulator Setup
The goal with our research was to build a low-cost simulator that allows to test new control concepts and design ideas in a fast and prototypical way. Therefore, it was of importance for us that the real and virtual world can be controlled and seen at the same time. We built a mixed reality simulator that allows to steer an excavator over an obstacle course by using a conventional joystick and wireless keyboard. Our mixed reality simulator setup consists of a head-mounted projection display, a room coated with a reflective cloth, input controllers mounted on a chair and software to drive and display the simulation.
3.1 Head-Worn Projection Display
Our head-worn projection display system, as shown in Fig. 1 consists of several off-the-shelf hardware: (1) a stripped-down laser pico projector, SHOWWX+ from Microvision, Inc., with external battery pack (2) a Samsung S4+ smartphone, (3) retro-reflective cloth covering the walls of our simulator room (can be seen Fig. 2), and (4) a headband with 3D printed housing holding the equipment.
A pico projector acts as a light engine in our head-mounted projection display. This has the ability that users who are wearing the prototype do not suffer from any key distortion effect, even when the reflective cloth used as a screen is distorted or not perfectly flat. This enables for a faster screen or CAVE-like setup by still allowing a clear and distortion free image. This lies in the optical abilities of the laser projector. The maximum native resolution of the projector is \(848px\times 480px~@60\,Hz~vsync\) in size by a light emission of 15 lumen. To increase the image size, we attached a \(180^{\circ }\) fisheye lens which allows for a field of view of roughly \(83.6^{\circ }\) x \(47.5^{\circ }\).
For our mixed reality application, we used the gyroscope sensor of the smartphone to look around in the digital environment and connected the pico projector through a MHL adapter to the smartphone. The overall system, has an expected uptime of 3 – 4 h.
3.2 Projection Room
The simulator room was coated with a high-gain retro-reflective cloth no. 6101 from RB Reflektör. The cloth is used to reflect the projected light back to the source and has a high light gain. This effect allows users standing close to the projector to basically see a very bright image. We placed the reflective cloth in a \(4\,\times \,3\) m room with a height of 3 m and also covered the floor in front and around the user. In this room we placed a rotatable chair and mounted a joystick and bluetooth keyboard on the chair. The setup of the room and the chair can be seen in Fig. 2.
3.3 Architecture and Software Description
Figure 3 provides a general overview of the technical components and the implemented architecture of our industrial simulator setup. The smartphone holds the digital simulator environment, developed in the Unity game engine version 5.2.3. The built-in gyroscope sensor of the smartphone is used to be able to look around in the digital environment with natural head movements. Moreover, we use the smartphone as a connection hub for the input devices and as a computation unit to show the digital environment through the connected projector. As joystick we used a Thrustmaster T.16000M which is connected to an Intel Compute Stick that runs a self-developed program converting joystick events into an XML format which is then sent via WIFI to the smartphone. Keyboard inputs were sent via Bluetooth to the smartphone. This data was thereafter used in the Unity engine to control the virtual environment.
4 Functionality Test
For our informal functionality tests and to get a first impression from users about our industrial simulator, we conducted a test with 21 users (4 female and 17 male). Three testers were in the range of 20–25 years, 9 users were in the range of 26–35 and 9 were over 35 years old. Only one user had driven an excavator before; 8 testers have not experienced a wearable projector or display before and 13 tried it before. Moreover, 8 out of 21 users tried a vehicle or industrial simulator before.
To evaluate, if our mixed reality prototype could be useful as a simulator, we also built a PC-based simulator that held the same digital environment and controllers (joystick and keyboard). Figure 4 shows the PC-based test setup.
The user tests were conducted on one day and each user was given an introduction to both prototypes, its functionality and a minute of time to try out the controls. Users started with the PC-based simulator and performed an obstacle course where instructions were given during the test. Thereafter, the users performed the same tasks again in the mixed-reality simulator. We chose this order so that users were able to get familiar with the controls and the tasks at hand in a more familiar PC-based simulator environment. When testing the mixed reality simulator, users already had an understanding on how to control the excavator and what the tasks were.
Driving the excavator was set to the W, A, S, D keys of the keyboard and steering the rest of the excavator was performed via the joysticks axes and buttons. Moving the lower excavator arm up and down was set to the vertical axis of the joystick, moving the turret of the excavator to the horizontal axis, moving the upper arm was controlled by twisting or turning the joystick around the z axis and finally two buttons on the top of the joystick controlled the bucket movement of the excavator. These controls do not fully resemble the controls of a real excavator. This was intentionally chosen, as new control and design ideas can be tested with our mixed reality prototype. The combination of choosing a keyboard and a joystick might be rather unconventional in a real vehicle but fits to our concept of exploring new control and design concepts for heavy machinery. This might especially be interesting as users and gamers are used to such controls. Nonetheless, we did not focus on exploring this aspect in this research further.
An overview of the obstacle course is given in Fig. 5. The general tasks were to drive along the track, over and through different obstacles without hitting objects or placed traffic cones. We placed 2 cube towers on the obstacle course where users had to look up and use the excavator to push down the top 2 cubes of each cube tower. This task was especially meant to test the controls of the excavator and to evaluate the difference between the two simulators. Completing one obstacle course in a simulator took about 5 min, the whole user test for one user took 15 min to complete.
The user tests were also videotaped as reference material and as another source of data collection to evaluate the user reactions and comfort or discomfort while using the prototype. In addition, a questionnaire was filled out by the users to help evaluating their experiences with both prototypes.
In the questionnaire we asked the users three for us essential questions: “Which of the simulators felt more realistic?”, “Which of the simulators felt more immersive?” and “Which of the simulators felt more natural to use and explore?”.
Figure 6 shows the results of these three questions. From the answers, we can see that for all three questions, the users are indicating that the mixed reality simulator offers a more realistic feel.
Moreover, we asked the users if they experienced a feeling of nausea or discomfort while using the mixed reality simulator. The answers were fairly distributed and showed that 42.86 % of the users had no such feeling, 23.81 % felt a light feeling of discomfort, 4.76 % had a mild feeling of discomfort and 28.57 % perceived a clear feeling of discomfort. These results are depicted in Fig. 7. Previous research has performed user tests with a car simulator using Oculus Rift DK2 VR glasses and tested 25 users [13]. They found that 84 % of their users felt motion sick in their tests. Our test results seem to show fewer issues with motion sickness. Nonetheless it needs to be tested with VR and even AR glasses in future research to solidify our findings.
Furthermore, we asked the user to rate their effort to complete the tasks in the two simulators. This indicated that it was slightly more difficult to complete the tasks in the mixed reality simulator, as can be seen in Fig. 8.
Testing our mixed reality simulator has shown that the general perception was positive and that users showed an indication of preference towards the mixed reality simulator in comparison to a PC-based setup. During the user tests, we observed that the users were quite focused on the tasks at hand and the, for them, rather unknown way of controlling an excavator. Our mixed reality setup supported the users, as the real world controls were visible and could be touched and tested naturally.
5 Conclusion
In this paper, we have shown a low-cost mixed reality simulator for industrial vehicles. The simulator was built from off-the-self equipment and allows for flexible scenarios, e.g. CAVE-like or in-vehicle setups. Our simulator uses a head-mounted projection display with a single 15 lm pico projector and does not require a cable connection to a stationary processing unit. Perceived images are focus-free, bright and distortion-free and the perceived field-of-view is approximately \(96^{\circ }\) in diagonal. The vision of the users is not occluded and no hardware is placed in the users’ field of vision. To evaluate our mixed reality simulator, we performed user tests with 21 participants and introduced a PC-based simulator using the same digital environment and controls as a baseline of our evaluations. The user test results indicated that participants had a more realistic, natural to use and immersive experience in the mixed reality simulator in contrast to the PC-based simulator. We observed that the ability to naturally look around in a CAVE-like large screen environment by still being able to see the real-world controls was beneficial to achieve these results.
In our view, we see the short set up time and low-cost of our mixed reality simulator prototype as a way to allow testing or developing prototypes and implementing design ideas for industrial vehicles or even other areas. Our tests were performed in a CAVE-like room but could even be installed inside a real vehicle cabin and combined with the controls of the vehicle. Therefore, we see our prototype as a basis for further research and a way to introduce simulators to different stages in designing and testing vehicular concepts.
References
Akşit, K., Kade, D., Özcan, O., Ürey, H.: Head-worn mixed reality projection display application. In: Proceedings of ACE 2014, 11th Advances in Computer Entertainment Conference. ACM (2014)
Bolas, M., Krum, D.M.: Augmented reality applications and user interfaces using head-coupled near-axis personal projectors with novel retroreflective props and surfaces. In: Pervasive 2010 Ubiprojection Workshop (2010)
Bretschneider-Hagemes, M.: Development of a new low cost driving simulation for assessing multidimensional task loads caused by mobile ICT at drivers’ workplaces. – Objective-fidelity beats equipment-fidelity? In: Yamamoto, S. (ed.) Human Interface and the Management of Information. Information and Knowledge in Context. LNCS, vol. 9173, pp. 173–179. Springer, Heidelberg (2015)
Brooks, J.O., Goodenough, R.R., Crisler, M.C., Klein, N.D., Alley, R.L., Koon, B.L., Logan Jr., W.C., Ogle, J.H., Tyrrell, R.A., Wills, R.F.: Simulator sickness during driving simulation studies. Accid. Anal. Prev. 42(3), 788–796 (2010). assessing Safety with Driving Simulators. http://www.sciencedirect.com/science/article/pii/S000145750900092X
Cakmakci, O., Rolland, J.: Head-worn displays: a review. J. Disp. Technol. 2(3), 199–216 (2006)
Christodoulou, S., Michael, D., Gregoriades, A., Pampaka, M.: Design of a 3d interactive simulator for driver behavior analysis. In: Proceedings of the 2013 Summer Computer Simulation Conference, pp. 17:1–17:8, SCSC 2013, Society for Modeling& Simulation International, Vista, CA (2013). http://dl.acm.org/citation.cfm?id=2557696.2557716
De Winter, J., Van Leuween, P., Happee, P.: Advantages and disadvantages of driving simulators: a discussion. In: Proceedings of Measuring Behavior, pp. 47–50. Citeseer (2012)
Firth, N.: First wave of virtual reality games will let you live the dream. New Sci. 218(2922), 19–20 (2013)
Freeman, M., Champion, M., Madhavan, S.: Scanned laser pico-projectors: seeing the big picture (with a small device). Opt. Photonics News 20(5), 28–34 (2009)
Guillaumée, M., Vahdati, S.P., Tremblay, E., Mader, A., Bernasconi, G., Cadarso, V.J., Grossenbacher, J., Brugger, J., Sprague, R., Moser, C.: Curved holographic combiner for color head worn display. J. Disp. Technol. 10(6), 444–449 (2014)
Harrison, C., Benko, H., Wilson, A.D.: Omnitouch: wearable multitouch interaction everywhere. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 441–450. ACM (2011)
Hua, H., Gao, C., Rolland, J.P.: Imaging properties of retro-reflective materials used in head-mounted projective displays (HMPDS). In: AeroSense 2002, pp. 194–201. International Society for Optics and Photonics (2002)
Ihemedu-Steinke, Q.C., Sirim, D., Erbach, R., Halady, P., Meixner, G.: Development and evaluation of a virtual reality driving simulator. In: Mensch und Computer 2015-Workshopband (2015)
Illusions, T.: Castar (2014). http://technicalillusions.com/castar/
Kade, D., Akşit, K., Ürey, H., Özcan, O.: Head-mounted mixed reality projection display for games production and entertainment. Pers. Ubiquitous Comput. 19(3), 509–521 (2015)
Kemeny, A.: From driving simulation to virtual reality. In: Proceedings of the 2014 Virtual Reality International Conference, p. 32. ACM (2014)
Kolasinski, E.M.: Simulator sickness in virtual environments. Technical report, DTIC Document (1995)
McGill, M., Murray-Smith, R., Boland, D., Brewster, S.A.: A dose of reality: overcoming usability challenges in vr head-mounted displays. In: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA 2015, p. 177, NY, USA (2015). http://doi.acm.org/10.1145/2702613.2732491
Microsoft: Hololens: a new way to see your world (2015). https://www.microsoft.com/microsoft-hololens/en-us/hardware
Mistry, P., Maes, P.: Sixthsense: a wearable gestural interface. In: ACM SIGGRAPH ASIA 2009 Sketches, p. 11. ACM (2009)
Mistry, P., Maes, P., Chang, L.: Wuw-wear ur world: a wearable gestural interface. In: Extended Abstracts on Human Factors in Computing Systems, CHI 2009, pp. 4111–4116. ACM (2009)
Olsson, M.I., Heinrich, M.J., Kelly, D., Lapetina, J.: Wearable device with input and output structures, 21 February 2013, US Patent 20,130,044,042
Politis, I., Brewster, S., Pollick, F.: Evaluating multimodal driver displays of varying urgency. In: Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Automotive UI 2013, pp. 92–99, NY, USA (2013). http://doi.acm.org/10.1145/2516540.2516543
Ranta, P.: Added values of forestry machine simulator based training. In: International Conference on Multimedia and ICT Education, Linsbon, Portugal (2009)
Rolland, J., Thompson, K.: See-through head worn displays for mobile augmented reality. In: Proceedings of the China National Computer Conference (2011)
Rolland, J.P., Thompson, K.P., Urey, H., Thomas, M.: See-through head worn display (HWD) architectures. In: Chen, J., Cranton, W., Fihn, M. (eds.) Handbook of Visual Display Technology, pp. 2145–2170. Springer, Heidelberg (2012)
Sonoda, T., Endo, T., Kawakami, N., Tachi, S.: X’talvisor: full open type head-mounted projector. In: ACM SIGGRAPH 2005 Emerging Technologies, p. 32. ACM (2005)
Turner, P., Turner, S., Burrows, L.: Creating a sense of place with a deliberately constrained virtual environment, vol. 1, pp. 54–68. Inderscience Publishers, Geneva, Switzerland, May 2013. http://dx.doi.org/10.1504/IJCPS.2013.053554
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Kade, D., Wallmyr, M., Holstein, T., Lindell, R., Ürey, H., Özcan, O. (2016). Low-Cost Mixed Reality Simulator for Industrial Vehicle Environments. In: Lackey, S., Shumaker, R. (eds) Virtual, Augmented and Mixed Reality. VAMR 2016. Lecture Notes in Computer Science(), vol 9740. Springer, Cham. https://doi.org/10.1007/978-3-319-39907-2_57
Download citation
DOI: https://doi.org/10.1007/978-3-319-39907-2_57
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-39906-5
Online ISBN: 978-3-319-39907-2
eBook Packages: Computer ScienceComputer Science (R0)