Exploring user experience in "blended reality": moving interactions out of the screen

Video game players often learn to map their physical actions (e.g., pressing buttons) onto their on-screen avatars' actions (e.g., wielding swords) in order to play. We explored the experience resulted from eliminating this mapping by modeling the screen as a "window" through which virtual objects enter the player's physical space, and the player interacts with them directly without the mediation of an avatar. We define this interaction as "Blended Reality" (BR). We designed, developed, and evaluated a BR game prototype called "Apple Yard" in which the player was to use a wand to hit apples flying out of the screen. A camera was used to track the positions of the player's eyes and wand, and the 3D game scene was rendered accordingly to create the illusion of looking through a window. A user tests experiment conducted on this prototypes indicated BR's potential in camera-based entertainment.