AIE 2013 Research on Augmented Reality
Description of research area
Imagine anyone in this world can experience like Tony Stark in the "Iron Man" movies - seeing 3D virtual objects in the real world and interacting with them with one's hand.
This idea, so-called Augmented Reality (AR), is combining real-world and digital information together for an enhanced user experience and can have future applications in all areas of our lives: education, science, entertainment, business and manufacturing industry etc. Currently the mobile phones and wearable computing industries are driving the development of augmented reality, however, more businesses will be leveraging the multitude opportunities in AR to enhance customers' experience with better products / services, improve efficiency and reduce costs.
Unlike the virtual reality which creates immersive, computer-generated environments, Augmented reality is close to the real world and changing the way we view our surroundings. Picture yourself walking or driving down the street. With augmented-reality displays, which will eventually look much like a normal pair of glasses (e.g. Google / Meta Glasses), informative graphics will appear in your field of view, and audio will coincide with whatever you see. These enhancements will be refreshed continually to reflect the movements of your head. In fact, similar devices and applications already exist, particularly on smartphones like iPhone, Android phone etc.
With the help of advanced AR technology (e.g. adding computer vision and object recognition) the information about the surrounding real world of the user becomes interactive and digitally manipulable. Artificial information about the environment and its objects can be overlaid on the real world. I believe Augmented Reality will be the next generation of computing.
This idea, so-called Augmented Reality (AR), is combining real-world and digital information together for an enhanced user experience and can have future applications in all areas of our lives: education, science, entertainment, business and manufacturing industry etc. Currently the mobile phones and wearable computing industries are driving the development of augmented reality, however, more businesses will be leveraging the multitude opportunities in AR to enhance customers' experience with better products / services, improve efficiency and reduce costs.
Unlike the virtual reality which creates immersive, computer-generated environments, Augmented reality is close to the real world and changing the way we view our surroundings. Picture yourself walking or driving down the street. With augmented-reality displays, which will eventually look much like a normal pair of glasses (e.g. Google / Meta Glasses), informative graphics will appear in your field of view, and audio will coincide with whatever you see. These enhancements will be refreshed continually to reflect the movements of your head. In fact, similar devices and applications already exist, particularly on smartphones like iPhone, Android phone etc.
With the help of advanced AR technology (e.g. adding computer vision and object recognition) the information about the surrounding real world of the user becomes interactive and digitally manipulable. Artificial information about the environment and its objects can be overlaid on the real world. I believe Augmented Reality will be the next generation of computing.
Goals of research
The goal of this research work is to understand the fundamental of Augmented Reality (AR) technology and demonstrate how one can develop practical applications using the technology. In AR, we can simplify it by grouping it into two categories: Geolocation AR, Marker-based AR:
Geolocation AR is used in conjunction with the mobile device's GPS capabilities and supply information on the mobile device's display based on location. A good example can be the Google Map application that utilises the mobile device's GPS and compass to display information for nearly businesses on top of the camera's view.
Geolocation AR is used in conjunction with the mobile device's GPS capabilities and supply information on the mobile device's display based on location. A good example can be the Google Map application that utilises the mobile device's GPS and compass to display information for nearly businesses on top of the camera's view.
Marker-based AR can be used on a wearable device (e.g. Google Glass), a mobile device or a PC with a webcam, but does not use geolocation. Instead, an object is used as the point of reference for the AR application. The object can be a special kind of printed image, a page in a book, or even a 3D object.
For this research, Marker-based AR is focused as it is based on Computer Vision and Object Tracking which can be quite complex but high in demand. A marker can be a specialised barcode, 2D image or 3D object that is linked to the AR application. When the marker is placed in front of the mobile device camera or webcam, it acts as a "trigger" to retrieves the data via the AR application or Internet connection and also becomes the reference coordinate for the AR application that is displayed on a AR eyeglasses, mobile device etc.
For this research, Marker-based AR is focused as it is based on Computer Vision and Object Tracking which can be quite complex but high in demand. A marker can be a specialised barcode, 2D image or 3D object that is linked to the AR application. When the marker is placed in front of the mobile device camera or webcam, it acts as a "trigger" to retrieves the data via the AR application or Internet connection and also becomes the reference coordinate for the AR application that is displayed on a AR eyeglasses, mobile device etc.
Details of proposed implementation
Because the technology behind Augmented Reality is based on Computer Vision concepts including trough image processing using filters, convolutions, histograms, contours, segmentation, object tracking, camera calibration and 3D vision. To recognise objects, input image data is matched within a database running in the background and then combines "recognised" data with content that has been related to it. The basic idea of AR is to superimpose graphics, audio and other sensory enhancements over the real world environment in real time. To understand well on these concepts and algorithms can be huge and quite complex, my research will rather focus on the basics by implementing some AR prototypes using 3rd party AR software technology:
- Control a 3D fire truck via Hand and Body motion detection using Microsoft Kinect, Unity 3D
- Control a mobile device's touch screen via Hand motion and Finger gesture detection using Leapmotion, Unity 3D, VNC Server / Client
- Control a 3D Tank on mobile device via AR 2D Marker tracking using Metaio SDK, Unity 3D
- Control different 3D Tanks on multiple mobile devices in network mode via AR 3D Marker tracking using Metaia SDK, Metaio Point Cloud Generator, Unity 3D
Control a 3D fire truck via Hand and Body Motion Detection
This prototype demonstrates the control on a 3D fire truck via the Kinect device by detecting hand position. The truck will be running on a slow constant speed on a predefine track. The player will only have limited steering angle to change the steering direction while routes cannot be changed. The player also allows to alter the speed by a “on-screen virtual” gas paddle. Besides, I have also captured the player by Kinect and apply green screen special effect to key the player full body motion image into the provided 3D virtual scene in real time.
The Kinect Wrapper Package for Unity3D has been used to integrate the Kinect device with the Unity 3D. It has I have used The implementation is used by in integration Unity 3D
The Kinect Wrapper Package for Unity3D has been used to integrate the Kinect device with the Unity 3D. It has I have used The implementation is used by in integration Unity 3D
Control Mobile device in the Air via Hand and Finger Gesture
This prototype demonstrate the ability of using one's finger to interact with 3D virtual objects through viewing from the Meta AR glasses using the hand and finger gesture recognition sensor.
I have setup VM Client / Server and coded in Unity3D to show the on-screen display of an Android / iPad on the AR glasses. Although there were still some calibration errors during the testing phase, I have experimented the Leapmotion as well to emulate the finger gesture detection and implemented another prototype.
For the Leapmotion experiment, the touch and swipe detection are quite sensitive as the finger might touch the wrong button if not moving carefully and steadily, however, the task of controlling mobile device using finger gesture in the air is doing very well for the prototype.
I have setup VM Client / Server and coded in Unity3D to show the on-screen display of an Android / iPad on the AR glasses. Although there were still some calibration errors during the testing phase, I have experimented the Leapmotion as well to emulate the finger gesture detection and implemented another prototype.
For the Leapmotion experiment, the touch and swipe detection are quite sensitive as the finger might touch the wrong button if not moving carefully and steadily, however, the task of controlling mobile device using finger gesture in the air is doing very well for the prototype.
Control a 3D Virtual Tank in the Real World via ID Marker
This prototype demonstrate the ID marker tracking for simple AR mobile application running in Unity3D. As shown in the picture, the 3D virtual tank shows only when the ID marker is recognised by the camera. The tank and its turret can be controlled by touchscreen buttons.
Using the Metaio AR SDK, it can configure up to 512 different markers with no performance penalty. The ID markers were also extended to picture markers which can configure about 10-30 Markers on a modern mobile device.
In Metaio, two different tracking modes (i.e. tracking quality) are used for ID markers. The "Fast" mode uses a fixed threshold offset and should only be used with no varying lighting conditions. However, the "Robust" mode adapts the given threshold offset constantly over time for a maximum number of search iterations, thus it can be used also in situations where lighting conditions are subject to change.
Additional parameters are the size (in mm) and the matrix ID of the marker to be used.
Using the Metaio AR SDK, it can configure up to 512 different markers with no performance penalty. The ID markers were also extended to picture markers which can configure about 10-30 Markers on a modern mobile device.
In Metaio, two different tracking modes (i.e. tracking quality) are used for ID markers. The "Fast" mode uses a fixed threshold offset and should only be used with no varying lighting conditions. However, the "Robust" mode adapts the given threshold offset constantly over time for a maximum number of search iterations, thus it can be used also in situations where lighting conditions are subject to change.
Additional parameters are the size (in mm) and the matrix ID of the marker to be used.
Control a 3D Virtual Tank in a Real World MMO Game via 3D Tracking (SLAM)
This prototype demonstrate the Marker-less 3D tracking (SLAM) for AR mobile application running in Unity3D. As shown in the picture, the 3D virtual tanks display only when the REAL miniature buildings are recognised by the camera. These REAL miniature buildings are used to generate 3D Maps and stored digitally in the Metaio Toolbox after successfully tracking the real-world environment. Next we can use ID Marker to align the scale and rotation of the 3D Map with the real-world.
I have created this network multi-player mobile game where each player can choose to be either on the Red or Blue team, controlling his own tank and shooting all the enemy tanks.
This virtual battlefield of tanks is a very interesting experiment to see multi-players can experience an high-intensity game in the real world environment.
I have created this network multi-player mobile game where each player can choose to be either on the Red or Blue team, controlling his own tank and shooting all the enemy tanks.
This virtual battlefield of tanks is a very interesting experiment to see multi-players can experience an high-intensity game in the real world environment.
Application of the proposed research to the relevant industries
During implementation of above Augmented Reality prototypes, I have found many possible applications in various fields including marketing, entertainment, product development and production, medicine, manufacturing etc, especially the industries which rely heavily on brand awareness, explanation on service.
AR in Marketing
As Augmented Reality has been a while in the industrial sector, marketing and sales has recently embrace the technology. Brand awareness, product interaction and the overall product experience can be enhanced using AR to superimpose additional information onto the product, its packaging or marketing materials. In result, AR helps to attract the customer and to preview the benefits of product or service, building a relationship between the brand and the customers.
Besides, AR can provide information about the company or its product based on the location the customer is situated for a dealer or store locator. With GPS and compass capability on the mobile device, information about local points of interest such as reviews or ratings can be viewed by the customers.
While the print publications have the high user acceptance for centuries and the advantage of concrete reading experience, AR helps to enrich any print media (e.g. brochures, catalogues, ads, flyers, business cards or entire billboards) with 2D or 3D information as well as up-to-date content that can be delivered post-printing.
Apart from using the print media as a marker for AR content, customers can point the mobile device at the product and experience hidden content related to the brand / product. This kind of marker-less image tracking / object tracking provide a greater level of personalised interaction with the customers.
For example, the furniture retailer, IKEA, have recently created a mobile app in order to present their new catalogues. AR allows shoppers to place virtual furniture into their own living spaces in a few simple steps. By browsing through a collection of furnitures, users can place items in their own surroundings and arrange them till they are satisfied with the selection and placement. Next the same application will link the shoppers to an e-commerce shopping cart for making the purchase. These kind of greater product involvement results in much better customer satisfaction and "gamification" and will help the company to understand better on customers' behaviour and interests.
As Augmented Reality has been a while in the industrial sector, marketing and sales has recently embrace the technology. Brand awareness, product interaction and the overall product experience can be enhanced using AR to superimpose additional information onto the product, its packaging or marketing materials. In result, AR helps to attract the customer and to preview the benefits of product or service, building a relationship between the brand and the customers.
Besides, AR can provide information about the company or its product based on the location the customer is situated for a dealer or store locator. With GPS and compass capability on the mobile device, information about local points of interest such as reviews or ratings can be viewed by the customers.
While the print publications have the high user acceptance for centuries and the advantage of concrete reading experience, AR helps to enrich any print media (e.g. brochures, catalogues, ads, flyers, business cards or entire billboards) with 2D or 3D information as well as up-to-date content that can be delivered post-printing.
Apart from using the print media as a marker for AR content, customers can point the mobile device at the product and experience hidden content related to the brand / product. This kind of marker-less image tracking / object tracking provide a greater level of personalised interaction with the customers.
For example, the furniture retailer, IKEA, have recently created a mobile app in order to present their new catalogues. AR allows shoppers to place virtual furniture into their own living spaces in a few simple steps. By browsing through a collection of furnitures, users can place items in their own surroundings and arrange them till they are satisfied with the selection and placement. Next the same application will link the shoppers to an e-commerce shopping cart for making the purchase. These kind of greater product involvement results in much better customer satisfaction and "gamification" and will help the company to understand better on customers' behaviour and interests.
AR in Product Development and Production
Augmented Reality technology has enormous potential for product design and prototyping, since it takes a lot of time, money and material to produce a one-time-use prototypes and different departments having different goals must collaborate well in the creative design process.
For example in the automotive industry, instead of producing physical prototypes, AR enables companies to use virtual prototypes and perform 3D inspections, CAD comparison, dimensional analysis, reverse engineering and more. The AR technology nowadays can achieve very precise digital prototypes, replacing the construction of physical prototypes. These digital prototypes can simulate how well they can be fitted into the newly developed product. In other words, production and design errors are detected early, helping the engineers to reduce the amount of prototypes created and easier for various participants to understand how it works.
Augmented Reality technology has enormous potential for product design and prototyping, since it takes a lot of time, money and material to produce a one-time-use prototypes and different departments having different goals must collaborate well in the creative design process.
For example in the automotive industry, instead of producing physical prototypes, AR enables companies to use virtual prototypes and perform 3D inspections, CAD comparison, dimensional analysis, reverse engineering and more. The AR technology nowadays can achieve very precise digital prototypes, replacing the construction of physical prototypes. These digital prototypes can simulate how well they can be fitted into the newly developed product. In other words, production and design errors are detected early, helping the engineers to reduce the amount of prototypes created and easier for various participants to understand how it works.
AR in Entertainment
Video game companies are quickly exploring the Augmented Reality technology and have been releasing games on the mobile device / their handheld consoles.
Some of the most interesting AR games I have seen so far are created for Playstation Vita system. These AR games require the use of proprietary AR cards to locate the table position so that virtual game objects can be placed on top. Some of their AR games are very creative and the object tracking performs very well as game objects often follow the real-world coordinates even the player move the device around suddenly.
Video game companies are quickly exploring the Augmented Reality technology and have been releasing games on the mobile device / their handheld consoles.
Some of the most interesting AR games I have seen so far are created for Playstation Vita system. These AR games require the use of proprietary AR cards to locate the table position so that virtual game objects can be placed on top. Some of their AR games are very creative and the object tracking performs very well as game objects often follow the real-world coordinates even the player move the device around suddenly.
Another successful Augmented Reality game of this year 2013 is Ingress which is a real time massively-multiplayer online game created by Google. Player take a side -- Enlightened or Resistance -- which battle against each other over "Exotic Matter", which is seeded throughout actual locations around the world. The gameplay consists of establishing "portals" at the location of popular real-world landmarks such as sculptures, libraries, churches, and other heavily-trafficked buildings. The bigger the city, the more portals it tends to have, with densely populated areas the most portal-rich parts of the game map.
International links and fields are not uncommon, as Ingress has attracted enthusiastic players from cities worldwide amongst both young and old, to the extent that the gameplay is itself a lifestyle for some.
I think the game mechanics involve significant complexity beyond this basic setup, but the key is that Ingress involves physically traveling in the real world to capture territories and collect virtual objects to score points and advance in the game, which is most unique and challenging element of an AR game as compared to the traditional on-screen video game.
International links and fields are not uncommon, as Ingress has attracted enthusiastic players from cities worldwide amongst both young and old, to the extent that the gameplay is itself a lifestyle for some.
I think the game mechanics involve significant complexity beyond this basic setup, but the key is that Ingress involves physically traveling in the real world to capture territories and collect virtual objects to score points and advance in the game, which is most unique and challenging element of an AR game as compared to the traditional on-screen video game.
AR in Medicine
Imaging technology is widely used throughout the medical field, it is no surprise that this area is viewed as one of the most important for Augmented Reality applications. Based on the pre-operative imaging studies of the patient such as CT or MRI scans, the surgeon can plan for the surgery after viewing the internal anatomy.
Augmented Reality can be applied by visualisation of the path through the anatomy of the affected area. Firstly a 3D model is created from multiple views in the pre-operative study and the model is projected over the target surface to help the surgical procedure.
Visualisation of the path through the anatomy of the affected area is done by first creating a 3D model from the multiple views and slices in the pre-operative study. The model is then projected over the target surface accurately to enhance the performance of the surgical team and eliminate the need for the cumbersome stereotactic frames that are currently used for registration.
Another application for Augmented Reality in the medical domain is in ultrasound imaging. Using an optical see-through display the ultrasound technician can view a volumetric rendered image of the fetus overlaid on the abdomen of the pregnant women. The image appears as if it were inside of the abdomen and is correctly rendered as the user moves.
Imaging technology is widely used throughout the medical field, it is no surprise that this area is viewed as one of the most important for Augmented Reality applications. Based on the pre-operative imaging studies of the patient such as CT or MRI scans, the surgeon can plan for the surgery after viewing the internal anatomy.
Augmented Reality can be applied by visualisation of the path through the anatomy of the affected area. Firstly a 3D model is created from multiple views in the pre-operative study and the model is projected over the target surface to help the surgical procedure.
Visualisation of the path through the anatomy of the affected area is done by first creating a 3D model from the multiple views and slices in the pre-operative study. The model is then projected over the target surface accurately to enhance the performance of the surgical team and eliminate the need for the cumbersome stereotactic frames that are currently used for registration.
Another application for Augmented Reality in the medical domain is in ultrasound imaging. Using an optical see-through display the ultrasound technician can view a volumetric rendered image of the fetus overlaid on the abdomen of the pregnant women. The image appears as if it were inside of the abdomen and is correctly rendered as the user moves.
AR in Manufacturing
In manufacturing, Augmented Reality provides an efficient method of training staff for service and maintenance on new systems / machines by visualising data on the real-world objects for instruction scenarios. Multiple trainees can see the same overlay simultaneously, providing better understanding into the systems and fostering clear communication between colleagues. As a result, trainers and trainees benefit from understanding better about the complex products and processes as well as the ability to display elements and procedures that are internal or not visible from the outside of a component or machine.
When the maintenance technician approaches a new or unfamiliar piece of equipment, instead of going through several repair manuals they can put on an augmented reality display, the image of the equipment would be augmented with annotations and information pertinent to the repair. For example, the location of fasteners and attachment hardware that must be removed would be highlighted.
Another example of AR is in machine and production sites, complex and dedicate operations can be visualised precisely beforehand. An example scenario is that a bulky engine requires replacement, with several solutions for the use of limited space visualised and discussed between all parties before the project begins. The connection between the engine and gear is simulated with AR, spatial requirements for maintenance are evaluated and multiple concepts for the installations of exhaust gas treatment are visualised. This blended simulation of real world dimensions augmented with computer designs saves both time, costs and reduces the risk of mis-planning and costly later rework or improvements.
In manufacturing, Augmented Reality provides an efficient method of training staff for service and maintenance on new systems / machines by visualising data on the real-world objects for instruction scenarios. Multiple trainees can see the same overlay simultaneously, providing better understanding into the systems and fostering clear communication between colleagues. As a result, trainers and trainees benefit from understanding better about the complex products and processes as well as the ability to display elements and procedures that are internal or not visible from the outside of a component or machine.
When the maintenance technician approaches a new or unfamiliar piece of equipment, instead of going through several repair manuals they can put on an augmented reality display, the image of the equipment would be augmented with annotations and information pertinent to the repair. For example, the location of fasteners and attachment hardware that must be removed would be highlighted.
Another example of AR is in machine and production sites, complex and dedicate operations can be visualised precisely beforehand. An example scenario is that a bulky engine requires replacement, with several solutions for the use of limited space visualised and discussed between all parties before the project begins. The connection between the engine and gear is simulated with AR, spatial requirements for maintenance are evaluated and multiple concepts for the installations of exhaust gas treatment are visualised. This blended simulation of real world dimensions augmented with computer designs saves both time, costs and reduces the risk of mis-planning and costly later rework or improvements.
Schedule of milestones during research and implementation
Risk management assessment for potential problem areas
As we have realised many current uses of augmented reality, let's look at the technology's limitations and what the future holds.
Using the geolocation AR, GPS in only accurate to within 30 feet (9 meters) and doesn't work well at indoors. Even though the image recognition might be able to resolve some of its drawback, it seems none of AR products in the market are proven to have 100% unshakeable image recognition and tracking due to intensive CPU usage on calculating object tracking algorithms. I think the AR technology will take some more years to improve and mature enough to offer real value to the mainstream.
Currently people may not want to use their mobile device for AR because of the small screen on which to superimpose information. As a result, wearable devices like Google/ Meta Glasses will provide users with more convenient, expansive views of the world around them. In future, AR will be used to create an Augmented City by detecting arbitrary buildings around us and add information to them. With face and body detection and tracking on the device, you may be able to track people's face and put information and virtual assets on top of them. Or you can play an AR chessboard game on your real table wearing with your AR glasses, inviting your online friend, and playing on the tabletop in front of you.
The future of Augmented Reality is clearly bright, could it be the future of computing?
Using the geolocation AR, GPS in only accurate to within 30 feet (9 meters) and doesn't work well at indoors. Even though the image recognition might be able to resolve some of its drawback, it seems none of AR products in the market are proven to have 100% unshakeable image recognition and tracking due to intensive CPU usage on calculating object tracking algorithms. I think the AR technology will take some more years to improve and mature enough to offer real value to the mainstream.
Currently people may not want to use their mobile device for AR because of the small screen on which to superimpose information. As a result, wearable devices like Google/ Meta Glasses will provide users with more convenient, expansive views of the world around them. In future, AR will be used to create an Augmented City by detecting arbitrary buildings around us and add information to them. With face and body detection and tracking on the device, you may be able to track people's face and put information and virtual assets on top of them. Or you can play an AR chessboard game on your real table wearing with your AR glasses, inviting your online friend, and playing on the tabletop in front of you.
The future of Augmented Reality is clearly bright, could it be the future of computing?