Hands on with Microsoft’s HoloLens

One second, you’re looking at the flat surface of a real wooden table. Then, you’re gazing straight through it, past small trees, tiny confused zombies, and layers of earth into a deep hole filled with animated lava.

It’s not a peyote-induced hallucination, but a genuinely impressive illusion created byMicrosoft’s (MSFT, Tech30) new augmented-reality goggles, HoloLens.

In this case, it was a 3-D game of Minecraft taking place, on, over, under, behind and inside real furniture and walls. As you move your head and body around, the illusion moves completely with you in 3-D, remarkably affixed to the real world objects.

Microsoft previewed the brand new piece of hardware on Wednesday. HoloLens is Microsoft’s foray into the virtual reality market. In its unique spin on VR, Microsoft has developed goggles that offer an augmented — or “mixed” — reality experience. UnlikeFacebook’s (FB, Tech30) Oculus Rift, which completely blocks out the outside world to fully immerse the wearer in another reality, HoloLens keeps one foot (and both eyes) firmly planted in the real world.

The lenses of the goggles are transparent, your view of the space around you only selectively blocked by digital images that can mingle with real objects. For example, an interior designer can move around a real room and rearrange 3-D pieces of furniture too see how they will look, even placing a virtual vase on a real shelf.

The hololens

To manipulate something in the HoloLens world, you use hand gestures, which are picked up by cameras on the front of the device. The single wiggle of an index finger can drop a flag on the surface of Mars or select a color for your 3-D sculpture. The cursor is always at the center of your view, moving when you move your head. Microphones pick up voice commands that bring up menus.

The goggles are wireless and don’t need to be tethered to any device, so while they’re meant for home or office use, they wearer is able to roam freely. The speakers play “spacial” sound, so a noise might seem like it’s coming from behind or beside you, adding to the virtual reality experience.

Related: Microsoft’s insane holographic vision of the future

After an on-stage demonstration of the sleek, wraparound black glasses, Microsoft gave reporters four hands-on demos of the device at its Redmond, Washington headquarters. (Notably, the goggles I tested were not the finished product Microsoft demonstrated on stage — they had jumbles of exposed electronic elements connected to a box that hung around my neck).

The demonstration scenarios attempted to show real examples of practical uses for the technology.

Video games. Gamers tend to be enthusiastic early adopters of experimental immersive technology, and the stunning Minecraft demo seemed like the most natural and realistic use case. I do not usually play Minecraft but would absolutely start, and maybe forget to stop, if I had a HoloLens.

On Microsoft’s HoloLens augmented reality goggles, you can play Minecraft in the real world.

Customer service. A Skype call demo was to walk me through fixing a broken light. The person on the other end of the call could see what I saw (real wires, tools, a hole in the wall) and could annotate my view with colorful lines and arrows. She even doodled a little diagram on a patch of blank wall. Remote customer support is a commonly imagined use for smart glasses, including Google Glass. While a neat thing to have, it’s nothing a phone or PC running video chat wouldn’t be able to accomplish.

Design. HoloStudio is a creative app that is designed for anyone to easily tinker with, like a beefed up MS Paint. A palette of colors and shapes floats in front of the artist who can create simple or complex 3-D models. Final designs can be ordered online, and a printed version of your 3-D purple skyscraper will be delivered to your door.

Virtual reality. At NASA’s Jet Propulsion Laboratory, scientists regularly dig through images of Mars collected by the Curiosity Rover. They’ve collaborated with Microsoft to create a program that places them inside a 3-D rendering of Mars using HoloLens. Whether you look up or down, turn left or right, or walk forward or back, the landscape shifts accordingly so you feel like you’re taking a stroll on the red planet.

The Mars demo was one of the few times I wished I was using full virtual reality goggles. They had reasons for wanted to keep reality involved — JPL scientists are working not playing, and they need to use other tools like computers at the same time. But as impressive as it was, the mixed-reality version of Mars looked thin and desaturated.

Full and partial virtual reality look different because of the underlying technology. HoloLens doesn’t display images on any screen, but projects light directly onto the retina, imitating the way light travels from real objects. The augmented view is contained in a rectangle directly in front of your face. Oculus Rift shows its images on a screen in front of the eyes and extends farther out and around for a more encompassing view.

The virtual reality space is still mostly filled with buzz and not-ready-for-prime-time products. But VR has been anointed the next big thing by a tech industry hungry for something fresh and exciting to woo customers.

Overlaying digital 3-D creations onto the real world is cool, but are there practical uses are there for this type of technology?

Microsoft hopes so. HoloLens was by far the most interesting announcement at a daylong event for Windows 10, and one Microsoft believes can help get customers excited about its products again.

Windows 10 to get ‘holographic’ headset and Cortana

Microsoft has revealed that Windows 10 will bring its voice-controlled assistant Cortana to PCs.

It also unveiled a headset that it said would one day project the operating system over views of the real world.

In addition, the firm announced that the OS upgrade would be offered free of charge for devices running Windows 8, Windows 7 and Windows Phone.

The offer, which is limited to the Windows 10’s first year of release, may aid its adoption.

It marks a change in strategy from Microsoft’s previous policy of charging for major updates, and could help avoid a repeat of the relatively slow uptake of Windows 8.

One analyst suggested the firm had needed to renew interest in its ecosystem.

“Overall, we know that about only about 10% of computers are running Windows 8 and the adoption rate among companies is similar or lower,” said Frank Gillett, an analyst at the Forrester consultancy.

“Developers are not paying much attention to Windows for mass market consumer apps, and you could even argue that for enterprise software most of the energy is going into mobile apps for iPad and Android tablets.

“Windows 10 is in effect a huge invitation to software developers to write exciting, powerful applications that will draw consumers.

“My hunch is that they can succeed in getting a new generation of PC and tablet applications. The challenge is getting people interested in its phones.”

Holographic helmet

Microsoft’s chief executive Satya Nadella said the HoloLens headset represented a “magical moment” of “category creation” that developers lived for.

The wearable tech’s augmented reality see-through lenses represent a major leap forward over Google Glass and other existing eyewear – assuming the machine lives up to its on-stage demo, in which computer-generated elements appeared in the world surrounding the wearer.

The company said that the final version of the machine would not need to be linked to other devices to work, and should be released within Windows 10’s “timeframe”.

It revealed Nasa was already working with the kit, and said the US space agency hoped to start controlling its Mars rovers with a prototype version as soon as July.

The HoloLens system features a see-through lens, which the firm said would provide a more “personal” experience than other devices

Other demos involving the machine included the wearer:

• playing Minecraft with the video game’s graphics appearing over living room furniture
• seeing a Skype video appear as if it was taking place on a building wall
• creating a model of a drone, which she saw in front of her face while shaping it by moving her hands and giving voice commands

“The true innovation HoloLens delivers will be determined by developer commitment in the months and years to come,” commented Geoff Blaber from the CCS Insight consultancy.

“Windows 10 is a defining moment for Satya Nadella early in his tenure as CEO.

“The collaboration required to deliver HoloLens to market is an encouraging sign that he’s breaking down the silos that slowed innovation and stalled execution in recent years.”

Microsoft showed off an augmented reality version of Minecraft as part of its HoloLens demo

Cortana on PCs

Windows 10 brings the same operating system to devices of all sizes, rather than having different ones for PCs/tablets, mobile phones and the firm’s Xbox games console.

One of the key features that consumers will be able to use on PCs from “day one” is Cortana – the voice-controlled tool previously limited to Windows Phone handsets.

Microsoft executive Joe Belfiore demonstrated how the software could be told to:

• bring up Powerpoint presentation documents
• locate photos taken during a specific month
• dictate and send emails
• tell the user whether or not the weather forecast indicates they should wear a coat later
• show where their car is parked

In addition, Mr Belfiore showed how the software could be used to respond to requests that had been typed, rather than spoken into a PC.

Windows 8 had been criticised by some for placing so much focus on touch-centric commands rather than the mouse and keyboard.

But Mr Gillett said he thought the addition of Cortana’s voice controls could prove more popular.

Cortana will be able to make suggestions based on a user’s past behaviour

“Voice is a going to complement other ways of interacting with the computer not be a substitute,” he said.

“If you had to say every command instead of touching or clicking, then that would be annoying.

“But if you can quickly say to Cortana, for example, schedule lunch with my mum next Tuesday, then that is powerful.”

Browsing with Spartan

Mr Belfiore also highlighted one of the core benefits of Windows 10: a single app will run on multiple types of device, with the user interface reformatting itself to suit the machine it is running on, rather than having a different program for phones, tablets and PCs.

He showed how this meant more advanced versions of the firm’s popular Office programs Excel, Powerpoint and Word could be brought to handsets running Windows 10 than were possible under Windows Phone 8. He also demonstrated a new “universal” Photos app that collates and sorts pictures taken with different devices.

Mr Belfiore also confirmed reports that the firm’s Internet Explorer web browser was being replaced by a new program codenamed Project Spartan.

This will have Cortana built-in to allow voice commands and provide extra personalised information – such as directions to a restaurant whose website is being looked at.

The software also includes a new “noting mode”, which will let users scribble or type over a page and then share it with others.

“Project Spartan shows a new Microsoft that is not afraid to depart from legacy in order to deliver a better experience,” remarked Carolina Milanesi from Kantar Worldpanel Comtech.

Xbox chief Phil Spencer showed how Microsoft’s Surface Pro tablet could be used to play Xbox One racing game Forza Horizon 2

The new system also introduces the ability to stream Xbox One video games from the console to any other Windows 10 PC or tablet on the same wi-fi network.

This means that users will be able to play any game in their Xbox library on another device in another room of their home if the property’s main TV is being watched by someone else.

The facility is similar to the service Sony offers for its PlayStation 4 games machine that allows it to stream titles to Xperia phones and tablets and the PlayStation TV mini-console.

Many of the headline features of today’s announcements were widely expected.

Some – like voice dictation on mobile, a cloud-based music service and auto-enhance for photos – are strikingly similar to those already available on rival operating systems.

But others are genuinely innovative – quite aside from its awe-inspiring holographic ecosystem.

Gamers will appreciate making the traditional gaming experiences more social and cross-platform.

Heavy web users may be tempted by the collaboration and offline features of the Spartan web browser, as well as the deep integration of Cortana.

And the addition of this powerful natural voice assistant on the desktop is groundbreaking too.

But the case for voice interaction on PCs is not as well-proven as it is on mobile – and Cortana needs to be well executed, otherwise the tried and tested keyboard and mouse could still prove too tempting for users.

Windows 10’s user interface is designed to be easy to adopt for users of Windows 7 and Windows 8

The key question: will the combination of the familiarity of past versions of Windows and an enhanced feature set be enough to tempt users to upgrade?

Offering Windows 10 free for the first year is a huge incentive to kickstart adoption.

Windows 10 is well positioned to capitalise on the resurgence of PCs.

But Microsoft is hoping for more than this: it needs the OS to provide a much-needed boost for its mobile ecosystem – and in this regard Microsoft is betting that its universal approach to coding apps will be enough to tempt users away from rivals.

Developers: over to you.