The U.S. Army Research Laboratory (ARL) is the Army’s corporate, central laboratory. “Its diverse assortment of unique facilities and dedicated workforce of government and private sector partners make up the largest source of world-class integrated research and analysis in the Army.” [1] Col. Harold A. Buhl Jr. is ARL Orlando Simulation and Training Technology Center (STTC) and the ARL Institute for Creative Technologies (ICT) program manager.
How might you describe mixed reality in terms of examples with which a single individual might relate in ordinary life?
Col. Buhl: When we talk about mixed reality, we’re really talking about a spectrum of capabilities. So if you consider the live world out there as one end of that spectrum, and you consider a fully immersive, virtual world as the other end of the spectrum, in between you have ways to blend the virtual and the live. A common example of this blend is the first-down marker you see when you’re watching football on television. That first-down marker is obviously not on the physical field. It’s a computer-projected entity into a live streaming video.
Now, thinking about a future scenario, let’s say you had shopped for handbags online. The next day, you’re walking down the street wearing a head display, and you’re alerted that the bag you looked at online is actually right down the street in a nearby store. Your system knows where you are and says, “Hey, you were online looking at this handbag. It’s available two blocks down the road on the second story on the third table.” Just like we have navigation for our cars today, the wearable device of the future may show you exactly how to walk to the store and purchase the bag.
How do you imagine mixed reality might benefit the work of the military?
Col. Buhl: Across the virtual portfolio, there are many great technologies and capabilities for the military. The key is that they can help us immerse soldiers in a virtual world so they experience operations in a controlled environment before they see they for the first time in combat or other stressful live situations. We’re able to use mixed reality to train soldiers in an emergency situation, without putting anyone or anything at risk. Virtual or gaming environments can be very immersive and really make the stress level go up, the heart palpitate, and the trainee break a sweat. This is very valuable.
At the Army Research Laboratory, we have a very robust human-sciences research campaign looking at the neuroscience and psychology, to be able to better understand what and how much information to provide soldiers. Every person is different, and being able to understand what is important to that person and what is necessary for that person to complete their mission is key. We run the risk of overwhelming a soldier or a tank commander or a helicopter pilot if we provide every bit of information possible, so understanding the neuroscience and psychology at play is key. It brings us into that whole human-machine interface area, where there’s a robust set of research that we need to explore so that we are actually facilitating the mission, whether it be combat in a hostile area or some other operation.
The military sees tremendous potential for mixed-reality technologies outside of combat applications, as well. Disaster relief, for example, will certainly benefit from mixed-reality technologies. In flood disasters, for examples, you could see the extent of flooding, what the projected flood plains are, if the water is it going to be rising or lowered. All this information is important as you’re aiding in search and rescue. Mixed-reality technologies can provide awareness of just how safe the environment is so that you can facilitate the evacuation and rescue of people.
We’re currently working with the U.S. Department of Homeland Security (DHS) on simulations. Right now, they are primarily semi-immersive virtual. We did one scenario with DHS where a terrorist set fire to a hotel and the terrorist was still armed. The situation caused the first responders some confusion because the firemen refused to fight the fire since the gunman was still at large, and the police refused to go after the gunman because of the fire. So the two disciplines had to work through that situation and develop tactics of just how they could work together in that type of situation.
A lot of military investment in mixed reality has benefited the commercial side. Technologies you see today like flat-panel displays are a result of military investment. Twenty years ago, we had tank and helicopter simulators with cathode-ray tube screens trying to project a virtual environment. We realized that they were very expensive, used a lot of energy and were very bulky, so we pioneered technologies to make flat-panel displays. It’s a great example of commercial investment and military cooperation.
How do you see mixed reality evolving in your field?
Col. Buhl: Here at ARL Orlando, we’re focused on the training potential of mixed reality. We’re looking at what can we do for the soldier with advanced technology to provide the most accurate information to the Soldier. Meanwhile, our university-affiliated research center, the ICT at the University of Southern California, is working with the artificial-intelligence aspect of mixed reality.
We’ve found that most people are more trusting of an avatar than they are of a live person. So when it comes to treating things like post-traumatic stress, we found great success with computers using things like cameras and LiDAR (light detection and ranging). A computer can watch eye movement to see if you’re being evasive, watch head movement and even track skin temperature to see if you’re flush or not. The computer is able to gather verbal and non-verbal information and uses AI to address the root causes of post-traumatic stress.
A key thing that augmented reality and mixed reality can provide is additional information to ensure safety and awareness. Things like intelligent tutoring and adaptive training can get people to a higher level of understanding more quickly. Mixed-reality types of interfaces are allowing us to customize opportunities based on interest and aptitude areas so that we can provide more positive solutions for the good of mankind.
What role do you envision IEEE playing within mixed reality?
Col. Buhl: Organizations like IEEE play an important role in developing and reinforcing definitions and standards. This is important because governments and policies often struggle to catch up as new technologies come onboard. Sometimes, there’s an immediate urge to be very restrictive when these technologies are not well understood. I think it’s important to develop some basic standards and definitions that are focused on the value these technologies bring to the community. As these are created, organizations like IEEE can help prevent duplications. IEEE can provide sufficient competitive space but bound that space so that people are not developing Betamax versus VHS, for example.
For more information on these emerging technology areas, please watch “Mixed Reality—The Future of Our World,” see The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems’ “Ethically Aligned Design: A Vision for Prioritizing Human Well-being with Autonomous and Intelligent Systems” and get involved with IEEE Future Directions and the IEEE Standards Association (IEEE SA). Also, please visit the IEEE SA at Augmented World Expo (AWE) 2018, 30 May and 1 June 2018 at the Santa Clara (California) Convention Center.
[1] Retrieved 13 January 2018 from https://www.arl.army.mil/www/default.cfm?page=20.