The Future of Mixed Reality: Issues in Illumination and Shadows

Abstract

Military training, concept design, and pre-acquisition studies often are carried out in virtual settings in which one can experience that which is, in the real world, too dangerous, too costly, or even beyond current technology. Purely virtual environments, however, have limitations in that they remove the participant from the physical world with its visual, auditory, and tactile complexities. In contrast, mixed reality (MR) seeks to blend the real and synthetic. How well that blending works is critical to the effectiveness of a user's experience within an MR scenario. The focus of this paper is on the visual aspects of this blending or more specifically on the interactions between the real and virtual in the contexts of proper inter-occlusion, illumination, and inter-shadowing. This means that the virtual objects must react properly to changes in real lighting and that the real must react properly to the insertion of virtual lights (e.g., a virtual flashlight or a simulated change in the time of day). Even more challenging, virtual objects must cast shadows on real objects and vice versa. The proper casting of shadows is critical to military training, in that shadows often provide clues of others' movements, and of our own to others, long before visual contact is made. Realistic shadows can improve training greatly; their omission or the insertion of physically incorrect shadowing can lead to negative training. To be effective, visual realism requires all such interactions occur at interactive rates (30+ frames per second). Our research focuses on algorithmic development and implementation of these procedures on programmable graphics units (GPUs) found commonly on today's commodity graphics cards. The algorithms we develop are tailored to take advantage of the parallel pipeline architecture of GPUs. Our primary application is training of dismounted infantry for the complexities of military operations in urban terrain (MOUT).