Microsoft’s HoloLens explained: How it works and why it’s different

Has Microsoft suddenly pushed us into the age of “Star Trek” and “Minority Report”? For those confused about what’s actually going on with the company’s new head-mounted gadget, here’s the rundown.

Microsoft has a vision for the future, and it involves terms and technology straight out of science fiction.

But are we actually glimpsing the future? Yes and no.

Microsoft’s HoloLens, which the company unveiled at its Redmond, Wash., headquarters on Wednesday, is a sleek, futuristic headset with transparent lenses. You can see the world around you, but suddenly that world is transformed — with 3D objects floating in midair, virtual screens on the wall and your living room covered in virtual characters running amok.

Technology companies have long promised to bring us the future now, reaching ahead 5 or 10 years to try to amaze consumers with the next big breakthrough. Hollywood, on the other hand, has shown that tech in action (or at least simulations of it).

In “Minority Report,” for instance, Tom Cruise’s character used sweeping, midair hand gestures and transparent screens to do police work. Five years later, Apple unveiled the iPhone, and with it, a touchscreen operated by hand and finger gestures. Microsoft in turn served up its Kinect gesture-control device, which tracks people’s movements through space and feeds the data into an interface.

Going further, “The Matrix” showed hackers plugging computers into people’s brains to transport them to imaginary cities. And in “Star Trek,” computers used energy fields and visual tricks to create worlds people could touch and feel.

We’re not even close to those scenarios yet, but we’re taking tiny steps in that direction. Companies like Facebook, Google and Microsoft are now attempting to move that fiction toward reality, and the public is beginning to see those visions of tomorrow take form.

So how does the HoloLens measure up against other reality-altering gadgets?

What’s a HoloLens, and how does it work?

Microsoft’s HoloLens is not actually producing 3D images that everyone can see; this isn’t “Star Trek.”

Instead of everyone walking into a room made to reproduce 3D images, Microsoft’s goggles show images only the wearer can see. Everyone else will just think you’re wearing goofy-looking glasses.

Another key thing about HoloLens is what Microsoft is trying to accomplish.

The company is not trying to transport you to a different world, but rather bring the wonders of a computer directly to the one you’re living in. Microsoft is overlaying images and objects onto our living rooms.

As a HoloLens wearer, you’ll still see the real world in front of you. You can walk around and talk to others without worrying about bumping into walls.

The goggles will track your movements, watch your gaze and transform what you see by blasting light at your eyes (it doesn’t hurt). Because the device tracks where you are, you can use hand gestures — right now it’s only a midair click by raising and lowering your finger — to interact with the 3D images.

There’s a whole bunch of other hardware that’s designed to help the HoloLens’ effects feel believable. The device has a plethora of sensors to sense your movements in a room and it uses this information along with layers of colored glass to create images you can interact with or investigate from different angles. Want to see the back of a virtual bike in the middle of your kitchen? Just walk to the other side of it.

The goggles also have a camera that looks at the room, so the HoloLens knows where tables, chairs and other objects are. It then uses that information to project 3D images on top of and even inside them — place virtual dynamite on your desk and you might blow a hole to see what’s inside.

While playing a demonstration based on the popular game Minecraft, I tapped my finger on a coffee table in the real world. But what I saw was my finger chipping away at its surface. When I was done, I saw a lava-filled cavern inside.

That’s just a gimmick, but Microsoft said it indicates potential. HoloLens, Microsoft said, can transform businesses and open up new possibilities for how we interact.

I used the HoloLens to video chat with a Microsoft employee who was using Skype on a tablet. Her task? To help me rewire a light switch. She accessed a camera on the HoloLens to see through my eyes, then she drew diagrams and arrows where I was looking to show me what tools to pick up and how to use them.

Imagine how these tricks could be used to train pilots or guide doctors through complex operations.

Different from the Rift

So how about the Oculus Rift? Created by Oculus VR, a startup Facebook purchased for more than $2 billion in March 2014, the headset is considered the poster child of the blossoming virtual reality market.

From a distance, Oculus’ headset looks a bit like Microsoft’s HoloLens in that it’s a device worn on your head. But that’s where the similarities end. Whereas Microsoft wants to help us interact with the real world in new ways, Oculus wants to immerse us in an entirely new world.

To put it simply, the Rift headset is a screen on your face. But when it’s turned on, the images it produces trick your brain into thinking you’ve been teleported to a different world, like a starship out in space, or the the edge of a skyscraper. Oculus could, one day, take a more practical route, transporting you courtside to a live basketball game or to a sun-soaked beach to relax.

The goal for Oculus is to trick the user into believing they’re actually there — wherever it’s bringing you. That feeling is called “presence,” an ambition Microsoft’s HoloLens isn’t reaching for.

Enthusiasts say that moment, where your brain is tricked into believing you’re actually somewhere else, is magical.

“I’ve seen a handful of technology demos in my life that made me feel like I was glimpsing into the future,” wrote venture capitalist Chris Dixon, who helped lead investment firm Andreessen Horowitz’s funding in Oculus VR. “The best ones were: the Apple II, the Macintosh, Netscape, Google, the iPhone, and — most recently — the Oculus Rift.”

Oculus isn’t alone in its quest. Sony is attempting something similar with its Project Morpheus headset. Both have outspoken plans to use the technology to transform all manner of industries, starting with video games. But developers say it’s hard to get it right. The images need to be carefully connected to your physical movements without any delays. When they aren’t, consumers feel a form of motion sickness.

Same difference

Ultimately, these companies are on different roads to the same destination, which is trying to reimagine how we interact with computers. We’re all used to the mouse and the keyboard, and we’re learning to live with the glass screens of smartphones too. So far, each of these devices has been good enough to convey the information from a book or the scenes of a movie.

But Oculus, Microsoft, Google and others believe in a different, potentially more natural way to interact with our technology. These companies and the hardware they’re creating imagine a world where hand gestures, 3D images and images superimposed on reality are the next-generation tools for productivity, communication and everything else we use gadgets and the Internet for.

It sounds like science fiction, but if these devices work the way tech luminaries hope they can, such dreams may be reality sooner than we think.

Hands on with Microsoft’s HoloLens

One second, you’re looking at the flat surface of a real wooden table. Then, you’re gazing straight through it, past small trees, tiny confused zombies, and layers of earth into a deep hole filled with animated lava.

It’s not a peyote-induced hallucination, but a genuinely impressive illusion created byMicrosoft’s (MSFT, Tech30) new augmented-reality goggles, HoloLens.

In this case, it was a 3-D game of Minecraft taking place, on, over, under, behind and inside real furniture and walls. As you move your head and body around, the illusion moves completely with you in 3-D, remarkably affixed to the real world objects.

Microsoft previewed the brand new piece of hardware on Wednesday. HoloLens is Microsoft’s foray into the virtual reality market. In its unique spin on VR, Microsoft has developed goggles that offer an augmented — or “mixed” — reality experience. UnlikeFacebook’s (FB, Tech30) Oculus Rift, which completely blocks out the outside world to fully immerse the wearer in another reality, HoloLens keeps one foot (and both eyes) firmly planted in the real world.

The lenses of the goggles are transparent, your view of the space around you only selectively blocked by digital images that can mingle with real objects. For example, an interior designer can move around a real room and rearrange 3-D pieces of furniture too see how they will look, even placing a virtual vase on a real shelf.

The hololens

To manipulate something in the HoloLens world, you use hand gestures, which are picked up by cameras on the front of the device. The single wiggle of an index finger can drop a flag on the surface of Mars or select a color for your 3-D sculpture. The cursor is always at the center of your view, moving when you move your head. Microphones pick up voice commands that bring up menus.

The goggles are wireless and don’t need to be tethered to any device, so while they’re meant for home or office use, they wearer is able to roam freely. The speakers play “spacial” sound, so a noise might seem like it’s coming from behind or beside you, adding to the virtual reality experience.

Related: Microsoft’s insane holographic vision of the future

After an on-stage demonstration of the sleek, wraparound black glasses, Microsoft gave reporters four hands-on demos of the device at its Redmond, Washington headquarters. (Notably, the goggles I tested were not the finished product Microsoft demonstrated on stage — they had jumbles of exposed electronic elements connected to a box that hung around my neck).

The demonstration scenarios attempted to show real examples of practical uses for the technology.

Video games. Gamers tend to be enthusiastic early adopters of experimental immersive technology, and the stunning Minecraft demo seemed like the most natural and realistic use case. I do not usually play Minecraft but would absolutely start, and maybe forget to stop, if I had a HoloLens.

On Microsoft’s HoloLens augmented reality goggles, you can play Minecraft in the real world.

Customer service. A Skype call demo was to walk me through fixing a broken light. The person on the other end of the call could see what I saw (real wires, tools, a hole in the wall) and could annotate my view with colorful lines and arrows. She even doodled a little diagram on a patch of blank wall. Remote customer support is a commonly imagined use for smart glasses, including Google Glass. While a neat thing to have, it’s nothing a phone or PC running video chat wouldn’t be able to accomplish.

Design. HoloStudio is a creative app that is designed for anyone to easily tinker with, like a beefed up MS Paint. A palette of colors and shapes floats in front of the artist who can create simple or complex 3-D models. Final designs can be ordered online, and a printed version of your 3-D purple skyscraper will be delivered to your door.

Virtual reality. At NASA’s Jet Propulsion Laboratory, scientists regularly dig through images of Mars collected by the Curiosity Rover. They’ve collaborated with Microsoft to create a program that places them inside a 3-D rendering of Mars using HoloLens. Whether you look up or down, turn left or right, or walk forward or back, the landscape shifts accordingly so you feel like you’re taking a stroll on the red planet.

The Mars demo was one of the few times I wished I was using full virtual reality goggles. They had reasons for wanted to keep reality involved — JPL scientists are working not playing, and they need to use other tools like computers at the same time. But as impressive as it was, the mixed-reality version of Mars looked thin and desaturated.

Full and partial virtual reality look different because of the underlying technology. HoloLens doesn’t display images on any screen, but projects light directly onto the retina, imitating the way light travels from real objects. The augmented view is contained in a rectangle directly in front of your face. Oculus Rift shows its images on a screen in front of the eyes and extends farther out and around for a more encompassing view.

The virtual reality space is still mostly filled with buzz and not-ready-for-prime-time products. But VR has been anointed the next big thing by a tech industry hungry for something fresh and exciting to woo customers.

Overlaying digital 3-D creations onto the real world is cool, but are there practical uses are there for this type of technology?

Microsoft hopes so. HoloLens was by far the most interesting announcement at a daylong event for Windows 10, and one Microsoft believes can help get customers excited about its products again.