Change language:English
English
Português
The concept of Augmented Reality (AR) has already appeared more than 50 years ago, in 1962, but it is only nowadays that the bet on this technology has given clear results, with its applicability in the industry to prove advantageous for companies from various sectors. In this article, we look at the birth of augmented reality, as well as the existing technology, and for its various current applications, with particular emphasis on its use in the industry.
An augmented reality system has to combine the real world with virtual elements, it must be interactive, to run in real time, and the 3D objects must be in line with the real world. The virtual elements are associated to the sensorial modalities including: visual, auditory, touch, haptic and olfactory; however, most uses of RA technology are only for visual stimulation.
Augmented reality: a brief history
The first reference to the concept of Augmented Reality goes back to the year 1962 when Morton Heilig built a machine with immersive multisensory technology that he named Sensorama.
This machine offered an immersive multisensory RA experience as it was able to display 3D images, stereo sound, haptic sensations through body tilt and wind sensations and aromas (by way of example see the device developed by Tajuki Narumi[1]).
In 1968, Ivan Sutherland developed the first head mounted display (HMD [2]) which he called The Sword of Damocles, which is a device used on the head, or an integral part of a helmet, which has an optical display in front of an HMD Monocular) or from each eye (HMD Binocular).
Although the concept has been around for a long time, the term Augmented Reality was only created in 1992 by Boeing researcher Tom Caudell. He and his colleague David Mizell had the challenge of providing an alternative to the diagrams and marking devices used to guide workers on the company’s factory floor. The solution they developed was an HMD that could be used by workers and that displayed the diagrams and schematics of the aircraft and projected them on reusable plates, and the information to visualize could be altered through a computer.
Augmented reality: examples and approaches
AR became popular among the masses through the game Pokémon GO [4] which was released in July 2016. The game used the GPS and camera devices compatible and allowed players to capture, battle, and train virtual creatures, called Pokémon, which appeared on device screens as if they were integrated into the real world.
Fruit of the success of the game, that had the adhesion of millions of players, the term AR left of its smaller niche and entered the mouth of the masses. The technology used to augment the virtual elements (in this case Pokémons) in the real world was geolocation. However, the most common technology for this purpose is Computer Vision (CV).
Due to the GPS error [5], the positioning of the Pokémon has a notable mistake (usually greater than 6 meters). This error in the positioning precision in this context of the game Pokemon GO is not very important, but for AR uses in industrial applications this error value makes its practical application impossible.
Image 1: Pokémon Go
AR technology for greater accuracy uses CV through two main approaches:
- with markers, which are images printed with patterns optimized for camera recognition of the devices;
- without markers, in which recognition of real space is made through a three-dimensional reconstruction of that same space.
Augmented reality: applications
AR can be experienced through a large number of hardware devices such as smartphones, tablets, desktop or portable computers, smart glasses and HMDs.
- The approach to use between (1) and (2), and the quality of the virtual elements desired to increase, is what defines which device to use in each case.
- The approach (1) is supported by all such devices, but for the approach (2) normally only HMDs are used.
AR HMDs are high-cost devices and have a very specialized set of hardware, which is what is needed to do the three-dimensional reconstruction of real space. Currently, the best-known AR HMD is Microsoft Hololens, which is actually a Mixed Reality device.
Image 2: HoloLens: Microsoft Press tool pictures.
Virtual reality, augmented reality and mixed reality
Milgram [6] in 1994 defined something he called “Virtuality Continuum” [7]: a scale ranging from the completely virtual, virtuality (which is commonly known as Virtual Reality) and the completely real, reality.
Between these two extremes lies the Mixed Reality (MR), which is the combination of the virtual world and the real world, where an interaction between real and virtual objects is possible.
AR is a type of MR that is closer to the extreme of reality. More than the device recognizes the real space and only superimposes virtual elements to it, the MR intends, for example, that a virtual lamp is placed on a real physical table and be affected by the ambient light so that it is not possible to distinguish it, whether it is real or virtual.
The MR also intends to respond to technical challenges such as the occlusion of a virtual element by a real element, which is an important characteristic so that the integration of the virtual with the real is perfect. In addition, it is intended with MR that it is possible to manipulate virtual objects. Microsoft Hololens allows you to address these MR goals and that is why it is considered such a device.
Image 3: Reality–virtuality continuum
Augmented reality goggles: the best devices
One of the most anticipated MR devices is the Magic Leap One [8] which with its “Lightfield Photonics” technology, that promises to revolutionize the world of AR and MR. The technology promises to “generate digital light at different depths and combine perfectly with natural light to produce realistic digital objects that coexist in the real world.” The idea is that this advanced technology allows our brain to naturally process digital objects in the same way as real-world objects, in order to make smart glasses more comfortable to use for long periods of time.
The company Magic Leap, despite not even have launched one only device in the market, has already received a large investment (of 2.3 billion dollars) from several important investors such as Google, Alibaba, JP Morgan and Warner Bros among others [9]. This is generating a great expectation, that there will be a huge technological revolution in the AR area.
An AR device very targeted for the industrial market is called Daqri SmartGlasses [10], a pair of glasses that have a high price.
Similar to these, but at a more affordable price, are the Moverio Pro BT 2000 [11] from Epson and the Vuzix M300 [12]. These types of AR devices are called smart glasses because they are more compact and have less processing power. These devices are useful for presenting 2D information and usually have a basic Android operating system such as mobile devices.
A famous example of this type of smart glasses was the Google Glasses that were discontinued [13] for the consumer market in January 2015, but they had a relaunch in the business market with the company Glass [14].
Advantages of augmented reality in the industrial context
More recently, the industry has finally begun to realize that the application of AR is an innovative and differentiating factor in an increasingly competitive market. Its applications can be immense, as well as the advantages derived therefrom:
- better visualization of scenarios;
- better decision-making;
- better organization;
- better training;
- higher job security;
- less productive stops;
- cost reduction;
- innovation, etc.
Examples of applicability of augmented and mixed reality in the industry by CCG over time
CCG, through its Computer Vision and Interaction Graphics (CVIG) domain, over the years, has been developing multiple projects focused on this technology, across several areas and sectors of industry.
Each of these projects brought specific advantages to the business in question. Some of these projects and applications are the following examples.
Furniture
- VrInMotion e ArPad: visualization of furniture in a real situation, through an AR system. A usage concept that was later explored by IKEA [15].
History and heritage
- HeritageCARE: increasing the efficiency of historical building inspector through MR. This solution allows direct interaction with each parametric object in real time, accessing data from previous inspections and saving the new ones [16];
- Virtual Assistants: reproduction of 19th-century leather processing tasks by AR, in particular, in the place where it is installed (Centro Ciência Viva de Guimarães).
Footwear, clothing, fashion, shops
- Walkys – Virtual shoe tester: creating an interactive experience with AR, which allows any user to experience footwear virtually.
- Perfume design: I- SENSIS II: uniform diffusion in the physical space of aromas, synchronized with video images; resulting in a truly immersive multisensory experience, through a wireless version of the existing product.
- PT21: Shop of the future – Smart fitting room: virtual proof of clothing, through AR, developing the “Store of the Future”.
- FAMEST: immersive visualization of three-dimensional footwear models using AR techniques, creating tools and concepts for future footwear.
Industry and logistics
- Cognito: visualization of human activity, through a final interface with the user, by means of AR. This way, the capacity to capture, recognize and process human activity was increased.
- CMMS – Computerized Maintenance Management System: implementation of a new AR functionality capable of presenting in situ the maximum information available in the management system/database (monitored parameters, documentation, maintenance history, spare parts, etc.).
- UH4SP – Unified Hub for Smart Plants: Unified Hub for Smart Plants: development of a software architecture, oriented to services and technological solutions, incorporating IoT and Industry 4.0. AR functionalities include driver interface through a mobile device, focusing on in-plant navigation assistance and AR feature; and remote assistance through the exploration of new MR telepresence concepts.
In the scope of the CCG’s bet in the Mixed Reality, a demonstration was developed, together with SONAE MC, for the application in the context of hypermarket logistics. This application was featured in the TVI24’s “NXT – The Next Step” program. The solution aims to optimize the formation of the picking activity in SONAE MC and to increase levels of safety at work.
This project was part of the annual results demonstration of SONAE. It consists of the development of an MR solution for training in the preparation of orders.
As it can be seen, augmented reality can be applied in industry in a variety of ways, to multiple companies or institutions. Thus, it is increasingly usual and natural to absorb this type of technology in the industrial and corporate universe. It was a long path, but a beautiful path to cross, and a path that still has much to show.
The next level in terms of augmented reality and mixed reality is called the hybrid world.
Sources:
[1] https://www.geeky-gadgets.com/smell-enhanced-augmented-reality-creates-any-food-you-desire-10-08-2010/#sf474284
[2] http://www.decom.ufop.br/imobilis/tecnologias-hmd-para-realidade-aumentada-parte-1/
[3] T. Caudell and D. Mizell, “Augmented reality: an application of heads up display technology to manual manufacturing processes,” in System Sciences, 1992. Proceedings of the Twenty-Fifth Hawaii International Conference on, vol. ii, pp. 659–669 vol.2, Jan 1992.
[4] https://www.pokemongo.com/
[5] https://www.gps.gov/systems/gps/performance/accuracy/
[6] Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
[7] Milgram, Paul; H. Takemura; A. Utsumi; F. Kishino (1994). “Augmented Reality: A class of displays on the reality-virtuality continuum” (pdf). Proceedings of Telemanipulator and Telepresence Technologies. pp. 2351–34. Retrieved 2007-03-15.
[8] https://www.magicleap.com/
[9] https://www.crunchbase.com/organization/magic-leap/investors/investors_list#section-investors
[11] https://www.epson.pt/products/see-through-mobile-viewer/moverio-pro-bt-2000
[12] https://www.vuzix.com/Products/m300-smart-glasses
[13] https://www.wired.com/story/google-glass-2-is-here/
[15] https://www.youtube.com/watch?v=uaxtLru4-Vw
[16] http://ieeexplore.ieee.org/document/8124304/
About the author:
Nuno Sousa | Project manager @CCG, A.R.D. CVIG
Nuno Sousa has a bachelor’s and a master’s degree, with specializations in Application Engineering and Software Analysis and Design, both in Computer Engineering, at the University of Minho. In the last 10 years, he has been involved in the development and implementation of several projects in the areas of Augmented Reality, Software Development (Web and Desktop), Human-Computer Interaction, Mobile Computing, Home Automation, Information Systems, and Software Engineering.
