Seeing AI by Microsoft combines the power of AI and the cloud to deliver an intelligent app that narrates the world around you. The app helps the visually impaired and the blind see or get around or navigate their world with better ease. 

The Seeing AI App is for the blind or low vision community that have problems functioning in a world made for people with good vision. It provides productivity and ease of tools for labeling utilities, obstacle or object recognition, and reading printed text. While there are many apps out there that can do the same things, there is no one single app that can do everything it does.

How it Works

The app uses the phones camera and computer vision to analyse what is in front of the phone including objects, text, and people. It then relays the information to the user via the computer assistant that communicates through your phone’s speakers.

It is divided into several navigational channels corresponding with technologies that would make life easier for people with a visual impairment. 

These channels are Scene, Person, Currency, Product, Documents, and Short Text (the Scene and Person channels are still in development).

Short Text – The feature uses the phone’s camera to search for printed material. Once it finds it, it uses optical recognition software to translate the printed material to machine text that is read out to the user through the phone’s speakers. It can be great if you are going through your mail, reading menus, or need to find a hotel room number.

Document – The feature uses optical recognition software to read documents. All you need to do is aim your camera phone towards the page of a document and the software will provide prompts when the corners and edges are all visible to the camera lens. Once they are all visible, it will prompt the user to hold steady and then automatically take a photo that will be sent to the cloud servers for processing. Once it has been processed, it uses voiceover to read out the document, including the formatting.

Product – This comes with a barcode scanner that makes it easy to scan the barcode on products to identify the product and details such as ingredients, directions, and any other pertinent details. Locating the barcode can be quite the hustle which is why the Seeing AI beeps when it spots it and increases the rate of the beep when you move closer to the barcode. Once it is close enough to the code, it will snap the code and using AI will identify what the product is, including the details.

Person – Seeing AI uses object recognition software to recognise people’s faces. If you have a library of photos of family and friends, the app will tell you when one of those people is near when you point the camera at them. The app guides you to center the face of the person and then tells you to snap the picture. It will then estimate their emotion such as frowning or smiling. If the person is in your image database it will also read out their name once their face is in focus.

Currency – The app uses object recognition software to identify the denomination of the currency through and then reads it out to you (unfortunately this currently only recognises US dollars).

Scenes – This is an ongoing project in beta with experimental features that you have to turn on in settings. When you take photos the AI recognises the objects, then gives a description of it such as “probably a man sitting on a couch” or “street corner with signs”.

Recognise Images in other Apps – It uses object recognition software to identify objects in apps on your phone such as WhatsApp, Twitter, and Mail.

Other features include an audible tone for corresponding to the brightness of the surrounding; description of the perceived colour and reading out loud handwritten text. 

Where to download…

The Seeing App is currently only available on the App Store where you can download it onto your iOS devices for free.

For further information on the Microsoft Seeing AI app and for video demos of each function click here.

Further information on our DSA assessment centre.