Group 2 eReport

Introduction
“…AR allows the user to see the real world, with virtual objects superimposed upon or composited with the real world. Therefore, AR supplements reality, rather than completely replacing it. Ideally, it would appear to the user that the virtual and real objects coexisted in the same space…”[1]

This is the definition of Augmented Reality as supplied by Ronald Azuma, a research leader at Nokia Research Center and pioneer in the field. It quite succinctly describes a very alluring technology, the possibilities of which enraptured the members of this project group. Through our research, we each found our own niche of Mobile Augmented Reality that we decided to follow. Riley Charko is investigating the overall history of Augmented Reality as a whole. Jeff Perry explores the realm of visual AR, while Steve McPhail looks at location-based AR. Finally, Cody Brown is conducting research on the interactive aspects of AR.

History and Definition
Definition

Augmented reality is technology that visibly alters reality with image overlay using technology like object recognition and surface sensory to then project information or images or whatever your programs purpose is on a real time basis giving the user an augmented reality experiance.

History

Sensorama (1957)

The sensorama was designed by Morton Heilig, a visionary in augmented reality. This device was the first kind of device to project a virtual reality for the user with sight smell and even feeling as this machine could simulate riding a motorcycle [2]. The sensorama was Morton Heiligs first vision and step towards what augmented reality is today.

Telesphere Mask (1960)

The telesphere mask was another step towards augmented reality as we know it today. This mask was the first to use a widescreen and directional stereo sound [2].

After Morton Heilig

Many other developers in the field of augmented reality had developed other head mounted displays and other types of innovations in the work place like "Ivan Sutherland and Myron Krueger" [3]. Though the biggest step-up to real tangible augmented reality was Boeing's wiring software.

Boeing wiring software (1990)

Used in manufacturing airplanes this technology was the first kind of reality overlay visual aid for helping engineers put wires into airplanes and managing this task [4]. It displayed where the wires went and what type of wire was needed.

Visual AR
In a sense, all augmented reality falls under the “visual AR” categorization in some way, and this is because the sense of sight is the most easily manipulated, yet absolutely critical aspect of how we perceive the world around us. While it’s quite possible that other sensory manipulations could be employed (such as the sense of hearing) to provide a framework for AR to build off of, there exists an element of interpretation when working in those realms. Consider if certain environmental stimuli triggered various noises or vocal phrases created by an AR device. While this kind of thing has successfully been employed in things like GPS devices or interactive computer interfaces, these technologies are limited in their ability to assist the user. The fact is, when we see something concrete our minds are able to comprehend and analyze that data in milliseconds without any doubt or room for interpretation if the information is presented in a clear, concise manner. Furthermore, the limitations to what we can do are severely reduced, as we are able to use things like symbols and colors to create further levels of intuitive information, something we are already used to in modern society.

So with all that in mind, the question then becomes “how do we best create an interface that can interpret objects in our surroundings and display relevant, concise data regarding the subject to the user?” Inherently, that question is the foundation of all visual augmented reality devices and interfaces. The idea can even be taken a step further by having the AR interface respond in a way outside of simply displaying data (explored later on in this report) but for now let’s focus purely on reactions in a visual sense. While a wide variety of applications can be conceived, several consumer-based industry fields, such as tourism and advertising, have already been targeted and have begun to utilize visual AR elements.

How It’s Being Used Now
Let’s take a look at tourism first. Imagine you are in a foreign country, or city even, anywhere unfamiliar to you. You’re not too sure what kind of places you should visit, maybe you know of a couple major tourist sites from your preliminary research of the place, maybe you know someone who will show you around town, or maybe you have nothing at all to go off of. Using your AR interface (most likely your smart phone if this scenario takes place in present day) you scan your surroundings and discover several restaurants, shopping centers, historical places of interest, and even public washrooms in your near vicinity. On top of that you are also able to see specific street and building names (and with the proper technology, your present address as well, but more on that later in this report).

Such technology is not so out of the question, and in fact already exists. TripAdvisor’s iPad app now incorporates a feature they have called “virtual tours” which employs Google Street View images over-laid with things like the information we were looking at in our scenario, as well as access to a TripAdvisor’s data base of reviews and opinions collected by other users and even has the ability to find Twitter posts about certain tagged places (Figure 1)  [5] [6]

Unsurprisingly, the advertising industry has also seen the huge potential in AR and has begun to utilize it creatively. In essence, advertising AR works in the same vein as QR codes where you would have to point your device at a certain trigger and an AR interface would be created. However, where a QR code would take you to a website or maybe display a short movie, AR can do so much more. The creative scope of what can be done, and what has already been done, is simply too large to completely cover it all, but to give you an idea of what kind of things have been done, here are a few exceptional examples I have come across:

Lego’s “Digital Box” [7]
In a select few Lego stores in North America, they have begun using a kiosk that is equipped with a camera and a screen that displays what the camera sees. At first glance, this appears to be nothing more than just a video feed but if you hold up a box of a specific Lego product, say like a bus or helicopter, a 30 second 3D image of the finished, built product appears on the top surface of the box the user is holding (Figure 2). They can move the box around as much as they’d like and the image always remains on the same surface so user’s don’t have to worry about standing perfectly still to get the event to trigger or stay running. While this technology is not a standard across every Lego product, a wide variety of their newer products do have this feature included in them.

Pokemon Trading Cards [8]Pokemon.png
Total Immersion is a company that specializes in visual AR technologies in a variety of forms across a varied client base, but the work I found most fascinating by them was their Pokemon trading card AR software. Essentially what this program does (and hopes to do with all kinds of trading cards eventually) is it analyzes a Pokemon card placed in front of a camera and creates a moving 3D image of that specific Pokemon on the card’s surface, much like the Lego Digital Box. However, take two of these cards and face them towards each other and the Pokemon will begin fighting! This takes Pokemon card duels to an entirely new level without radically changing any of the interactive mechanics of the game itself.

The Artvertiser [9]
An idea created by a team of artists actually meant to undermine current conventions of advertising, the Artvertiser is an interesting piece of visual AR that overlays free art, whether it be a video or a still image, over top of an already created (generally corporate) ad (Figure 3). Target sights so far include popular areas such as Puerta del Sol in Madrid as well as Times Square in New York City, but the teams work has been displayed in several international art festivals as well. The objective is to challenge the concept of public space and who has the right to display messages on it, but all that aside it is still a very cool use of visual AR. The software runs on all common operating systems as well as Android-based phones and other select smartphones.

The list of original, unique applications goes on and on, but a common problem exists for them all: they require an external platform to be fully operate. While a lot of AR experiences are being tailored for smartphones, there still exists a sizeable chunk of software that is computer exclusive, which removes the concept of mobility in most cases. So the next large obstacle to conquer now becomes “how do we make these interfaces more seamless and mobile?” A team of researchers at the University of Seattle in Washington are on the verge of creating technology that will completely revolutionize how integrated visual AR will become in our lives…

The AR Contact Lens
The objective of the augmented reality contact lens is to have a small wearable interface that continually displays relevant information without the need to interact with a physical device. The lens in question would be powered by radio frequency from an external transmitter, which would also serve as the hub for information transmission, and create a heads-up-display (HUD) that would appear to hang about half a meter away from the user at all times [10]. This would make for a hands-free, totally seamless augmented reality experience.

However, several design issues and complications plague the contact lens from being used in a practical way presently. For instance, there is a roughly 1.5 cm of space on a standard contact lens for the team to work with, and considering cramming that space full of various components would remove the transparency of the lens that leaves very little room for development [10]. Then they have to consider how the lens in question is going to create a legible image for the user to read. They have arrived at using a small array of LEDs combined with a series of sub-lenses that focus different parts of the array to create an image, as simply creating an image on the surface of the lens would solve nothing seeing as the human eye cannot focus on a subject that is too close to the cornea [10]. Among all other difficulties, the contact lens has to be safe enough to be in constant contact with the human eye. Considering many of the components are built with substances that would be harmful to the eye, the whole lens has to be coated in a biocompatible material so serious injury or harm does not occur [11]. Fortunately, the method has proved to be effective as the team has recently used the current version of the contact lens on live rabbits for extended periods of time with no adverse effects. [10][12]

Although the technology may still be more on the horizon then just around the corner, the implications and applications of such a device could drastically change not only how AR works, but also how our lives in general. Consider how much of an impact mobile technologies have already had on our society, and how they continue to flourish. Will the contact lens be the next link in the chain, or will it create something totally new? Some would even go so far as to consider this human augmentation. But for now, it is merely something to keep in mind as we further discuss augmented reality.

Location-Based AR
A very popular use of Mobile Augmented Reality systems is to locate the user and provide relevant, context-based information about the surrounding area. This often takes form in AR Browsers. Through my research I have found four seperate projects that have been published within the past few years that all demonstrate how MAR can be used to calculate and utilize the user's location.

A System for GPS-aided Recognition-based User Tracking with AR [13]
The first of the research topics I delved into is the actual system used to make it work. It is comprised of a few steps, which are essentially universal for any location-based MAR system.

First, the user takes a picture of a building or other surrounding scenery. The device then breaks the image up into an overlapping grid so it can analyze each section individually in order to speed up the search. The GPS location of the device when the picture was taken is searched, and the closest "cluster" of points of interest that have been uploaded is returned to the device. These points of interest each have a photo associated with them. The image is analyzed algorithmically to figure out what grid sections are distinct, which are then matched the the photos already in the database. This information is then fed back to the user through the display of the device [13].

This provides a basic process for MAR systems - the important aspect of this device is not so much the device, but rather how it is used. The only device-specific features it has are a camera, internet-browsing capabilities, a viewscreen, and a powerful enough processor [13]- the first of these three are widely available on almost every device available now, and the power of these devices is greatly increasing [14].

CyPhone, A Visor-Mounted AR Cellular Phone [15]
CyPhone is a prototype for incorporating this sort of MAR technology into something similar to an already existing framework: cellular telephones. It uses a similar technique as the above mentioned device, but one of the two main differences is that the provided readout is located on a head-mounted display visor. This greatly enhances the reality part of augmented reality, as all the information is displayed on a live view of whatever the wearer is looking at as opposed to a still image on a device [15]. The other main difference is how the CyPhone derives its data. As opposed to image identification, it is purely GPS-based. While it will get information from a database, it is similar to what currently available GPS navigators provde - a GPS location for the POI. From there it will provide the user with an onscreen notification of where the POI is. A map can be displayed on the viewscreen, as well as routing information via arrows, a footprint path, or a digital avatar of a guide [15].

I personally think that this may very well be the future of mobile devices, or rather device accessories. Specifically, visors that connect to the device - ideally wirelessly with a protocol such as Bluetooth - and project data to the user visually. I believe it would cut down significantly the amount of distraction caused by having to look down at a device's screen, particularly in situations, such as driving, where this can be a huge liability. The possibility exists to not only have location information (maps, routing, etc.) but any other information from the phone displayed - incoming calls, text messages, scheduling.

Virtual Cable Volumetric HUD [16]
Continuing from the idea of GPS information being overlaid on the user's view as opposed to on a viewscreen that the user must redirect their attention to view, a California company has designed a device that takes this to an even more practical label. It is not quite mobile in the typical sense, in that it is not a portable handheld device, but can still be used from almost any point on the planet. It is a direct upgrade to current GPS navigator devices.

It is called the Virtual Cable, and consists of a circuit board, a laser beam, and a number of filters and screen, as seen below in Fig 1. The device, installed under the dashboard of the vehicle, calculates and generates the vehicle's GPS location, the route to the user's chosen location, and the nearest POIs - very similarly to how currently available GPS navigators operate [17]. The innovation involved in this device that caused it to win a prestigious award [16] is how the information is displayed to user: it is projected through a disguised mesh in the dashboard directly onto the windshield. This allows all the requisite information to be where the driver is already looking, as opposed to a screen that distracts from their view of the road. Not only is the map displayed unobtrusively on the windshield, information such as the vehicle's speed and direction as well as nearby POIs (which would be appear to the driver to be oriented on or near the physical locations).

The key selling point of the Virtual Cable system is the Virtual Cable itself: a line that appears above the path that the driver must take to reach his or her destination. It allows the driver to see at a glance exactly where they are going, without taking their eyes off the road. It is projected along with the other information onto the windshield, but some rather unique technology is incorporated into it - it has been designed so no matter what angle the user looks at it from, it appears to exist in the same relative location above the road ahead [16].

"Mirror World" Augmented Reality and Image Space [18]
The last product I found moves augmented reality into a social media context. Developed by a number of rsearchers at Nokia Research Center in Helsinki and Tampere [18], it is a concept of how to use MAR as opposed to a physical device. It uses a concept called a "mirror world", or a virtual space that maps directly onto the features of the real world. This allows users to post information onto the mirror world in association with real-world places and have this information displayed on a live view of the real world.

This brings a couple of possibilities to the field, as it emphasises user interactivity on a wide scale in the sense that what one user posts, all can see. An individual can take a picture at a location and have it save in a public space for others to see, abling them to not only see the picture, but the real life context of it. As well as this it can record a "trail" of where a user has been, which can then be shared with friends. This will allow them to, as put in the paper, "see not only where their friends are, but where they have been" [18].

I think this could be a new major trend in social networking, allowing people to be even more connected with each other. Facebook and the iPhone have similar uses, through the Locations feature and foursquare, respectively, and allowing this kind of social feedback builds and improves upon this.

Interactivity in Mobile AR
To give one a perspective into this topic, I will begin with a description of what it entails and some applications. Interactive Mobile Augmented Reality relates to the more modern development of Mobile Augmented Reality. This aspect deals with the software and the video capturing devices capability to interpret movement and patterns, as well as the presence of objects within the field of vision, rather than simple 2D overlays on the display [19] The idea behind this is to adapt Augmented Reality more to its surroundings, as the real world is dynamic in nature and constantly changing around us; that being said so must be the technology. By incorporating this intuitive nature and recognition into the software, it becomes more viable in a workplace or business environment, as well as expanding the possibilities for commercial use. For example, someone coordinating a presentation would have full control over his slides, animations, and videos simply by interacting via gestures towards a video capture device rather than controlling it directly, or a chairperson leading a meeting may control and interact with a 3D image to emphasize or explain an idea or concept.

While conducting my research, I discovered that on its very own, Interactivity in Augmented Reality is an encompassing and fascinating subject. Some of the most modern work I was able to find involved testing three different methods of sensory input via a test group of 18 individuals, of ages varying between 15 and 52 years [19]. To explain what exactly the study was trying to accomplish, we’ll go into some finer detail.

The main goal of the study was to determine the viability of three different main methods of sensory input in relation in to Mobile Augmented Reality through trials and challenges. The challenge was to select and translate an object in the field of vision, and place it somewhere else with the handheld device. This worked by interpreting movements, input, and gestures to coordinate x, y, and z axis movements and displaying it back on the viewscreen [19]. Some of the trials were simple, the object only having to move to another position within the same field of vision as where it originates, to more complex trials where the participant had to rotate and locate the destination. Using this standard for all the trials, the methods that which were being tested are as follows [19]:

Touch Screen

This feature is now available on all modern phones capable of augmented reality, and there is no better place to start. This involved physically touching the device with your fingertips while facing the device in the required position and angle. It was a fairly simple concept, touching the display in different ways, and in different combination with two fingertips, would translate the object on your viewscreen [19].

Device Managed

This concept delved into using your actual viewscreen as your tool for selecting and manipulating objects. It did this by placing a targeting reticule on your viewscreen which would represent the targetted area of interest when interacting with the environment. By hovering the reticule over an object that can be manipulated, a progress bar begins to fill, taking around 1.5 seconds, to confirm your selection. Once being selected, you may then manuever the object simply by moving the device itself [19].

Gesture Managed

The last method involved using your own hands to signal different commands by performing specific gestures while in view of the camera. By placing recognized colors on the tips of your fingers, the software is able to determine the very tips of your fingers, and analyze your gestures based on that. Selecting an object required almost the same method as you would on a touch screen by having to point at the object and then slide it around after selection to the desired location [19].

Each participant of the trial was asked to attempt to complete a series of interactions on a mobile device (smart phone or similar), and judged their performance based on [19]:


 * The amount of time to complete the challenge
 * The amount of time to complete the challenge


 * The amount of accuracy in completing the challenge
 * The amount of accuracy in completing the challenge


 * The amount of difficulty experienced by the user
 * The amount of difficulty experienced by the user


 * The effectiveness of the means on input
 * The effectiveness of the means on input


 * The amount of fun they had completing the challenge
 * The amount of fun they had completing the challenge

All of this data was obtained and observed throughout the trial. and was compiled so that it may bee critically analyzed by the coordinators. Their observations were that each individual sensory module presented its own strengths and weaknesses (see figure 1).



As you can see, there is a large difference in performance for each respective test. The thought behind this whole idea is to judge where each different method would have the best suited application, as each purpose requires customization to meet the specific needs of the task it is supposed to perform.

Another important aspect to take into consideration along side the amount of time it takes to complete the tasks is accuracy. The amount of accuracy that a potential input method is capable of is a large contributor to the usefulness in general (see figure 2). With an unreliable method, the physical interaction with the device to utilize the application becomes a burden rather than a compliment.

">

As was stated previously, being able to gauge the effectiveness of future methods of sensory input would have a great impact on how Mobile Augmented Reality is marketed and utilized in every aspect, and thus is a very important topic of research and analysis. This is an intriguing prospect, as it opens up some concepts which we are able to discuss. Having defined the utilities of different input methods, companies may begin to more rapidly adopt the technologies and software related to their needs in the industry, as well as moving forward the commercial market with applications which people found the most fun or interactive for their day to day operations and applications. To conclude this information, one might say that this type of research is what will really define how Mobile Augmented Reality will be used within each respective field, as well as how easily people are able to adopt new technology without having to worry about all the other aspects.

Conclusion
In conclusion, Mobile Augmented Reality is a very broad field with a wide range of potential applications. From its eveolution from Virtual Reality devices in the 1960s it has been developed into much more portable, intuitive interfaces that even now are widely available as smartphone applications. Other devices, such as the AR Contact Lens, are even more groundbreaking, as is the interactive technology that has been created. Some ethical issues are raised as to how easily these devices should be able to recognize what they "see" and how much of this information should be relayed to the user, of course. In closing, we do believe that this could potentially be a new mainstream technology.