The post Superheroes, puppies, hippos – and AI – are helping children with disabilities bridge language gaps By Emily Alhadeff appeared first on The AI Blog.

The post Superheroes, puppies, hippos – and AI – are helping children with disabilities bridge language gaps appeared first on The AI Blog.

Today’s intelligent assistants are full of skills. They can check the weather, traffic and sports scores. They can play music, translate words and send text messages. They can even do math, tell jokes and read stories. But, when it comes to conversations that lead somewhere grander, the wheels fall off.

“You have to poke around for magic combinations of words to get various things to happen, and you find out that a lot of the functions that you expect the thing to do, it actually just can’t handle,” said Dan Roth, corporate vice president and former CEO of Semantic Machines, which Microsoft acquired in May 2018.

For example, he explained, systems today can add a new appointment to your calendar but not engage in a back-and-forth dialogue with you about how to juggle a high-priority meeting request. They are also unable to use contextual information from one skill to assist you in making decisions from another, such as checking the weather before scheduling an afternoon meeting on the patio of a nearby coffee shop.

The next generation of intelligent assistant technologies from Microsoft will be able to do this by leveraging breakthroughs in conversational artificial intelligence and machine learning pioneered by Semantic Machines.

The team unveiled its vision for the next leap in natural language interface technology today at Microsoft Build, an annual conference for developers, in Seattle, and announced plans to incorporate this technology into all of its conversational AI products and tools, including Cortana.

Teaching context and concepts

Natural language interfaces are technologies that aim to allow us to communicate with computers in the same way we talk with each other. When natural language interfaces work as Roth and his team envision, our computers will understand us, converse with us and do what we want them to do, much like most people can understand a complex request that requires a few actions.

“Being able to express ourselves in the way we have evolved to communicate and to be able to tie that into all of these really complicated systems without having to know how they work is the promise and vision of natural language interfaces,” said Roth.

The natural language technology in today’s intelligent assistants such as Cortana leverages machine learning to understand the intent of a user’s command. Once that intent is determined, a handwritten program – a skill – is triggered that follows a predetermined set of actions.

For example, the question, “Who won today’s football match between Liverpool and Barcelona?” prompts a sports skill that follows the rules of a pre-coded script to fill in slots for the type of sport, information requested, date and teams. “Will it rain this weekend?” prompts a weather skill and follows pre-scripted rules to get the weekend forecast.

Since the rules for these exchanges are handwritten, developers must anticipate all the ways the skill could be used and write a script to cover each scenario. The inability of humans to script every possible scenario limits the scope and functionality of skills, explained Roth.

The Semantic Machines technology extends the role of the machine learning beyond intents all the way through to enabling what the system does. Instead of a programmer trying to write a skill that plans for every context, the Semantic Machines system learns the functionality for itself from data.

In other words, the Semantic Machines technology learns how to map people’s words to the computational steps needed to carry out requested tasks.

For example, instead of executing a hand-coded program to get the score of the football match, the Semantic Machines approach starts with people who show the system how to get sports scores across a range of example contexts so that the system can learn to fetch sports scores itself.

What’s more, machine learning methods then enable the system to generalize from contexts it has seen to new contexts, learning to do more things in more ways. If it learns how to get sports scores, for example, it can also get weather forecasts and traffic reports. That’s because the system has learned not just a skill, but the concept of how to gather data from a service and present it back to the user.

That’s missing in today’s intelligent assistants, which are programmed to do a list of isolated things that a programmer anticipated. The machine learning in these systems primarily focuses on words that trigger a skill, explained Microsoft technical fellow Dan Klein, a recognized leader in the field of natural language processing and a professor of computer science at the University of California at Berkeley.

“They aren’t focused on learning how to do new things, or mixing and matching the things they already know in order to support new contexts,” said Klein, who was also a co-founder and chief scientist at Semantic Machines.

Dynamic conversation

Since the Semantic Machines system can learn how to do new things, it can more easily engage in a dynamic conversation with a person, accessing and stitching together relevant content, context and concepts from disparate sources to provide answers, present options and produce results.

The Semantic Machines system also has a memory to keep track of the context in a conversation and so-called full duplex capability to talk and listen at the same time in order to keep the dialogue flowing.

“Everything you say is contextualized by what has come before so you can do more complicated things: you can change your mind, you can explore,” said Klein. “Moreover, once things get contextual enough, the notion of a skill begins to dissolve.”

That’s because the notion of skills confines interactions to silos of data whereas true conversation relies on connecting data from all over the place. The Semantic Machines technology orchestrates gathering data and accomplishing tasks on the backend while maintaining a fluid, natural dialogue with the user on the frontend.

Reshuffling your schedule to accommodate a high-priority meeting, for example, requires calendar data and directory data to determine who is free, when, as well as contextually relevant data such as the weather, nearby coffee shops and traffic to figure out where to meet and sit, and when to leave to get there on time.

“Once you start letting things evolve and connect contextually, the notion of a skill is way too limiting,” said Klein. “Getting things done involves mixing and matching.”

Building with natural language

At Build, Microsoft showcased a calendaring application using Semantic Machines technology that can make organizing your day with an intelligent assistant a more fluid, natural and powerful experience. The same technology can be applied to any conversational experience and will eventually power conversations across all of Microsoft’s products and services.

That will build on Cortana’s existing capabilities such as providing answers to questions, offering previews of your day and helping you across your devices from phone to laptop and smart speaker.

Once the technology is incorporated into Cortana, for example, it could make getting things done in Office more about what you need to do and less about accomplishing tasks in certain applications.

“We want it to be less cognitive load, less feeling like I have to go to PowerPoint for this or Word for that, or Outlook for this and Teams for that, and more about personal preferences and intents,” said Andrew Shuman, Microsoft’s corporate vice president for Cortana.

What’s more, added Roth, the technology will be made available through the Microsoft Bot Framework. His team is currently engineering a way for developers working in the framework today to migrate their existing data to the Semantic Machines-powered conversational engine when it is ready.

“As a developer you can start building these experiences yourself,” he said. “We can collectively move, on the basis of this technology, past this notion of skills and silos and simple handwritten programs into the kind of fluid Star Trek-like natural language interfaces we all want.”

Microsoft Build 2019 – Related links to conversational AI:

• Read about Microsoft’s acquisition of Semantic Machines
• Learn more about Semantic Machines

John Roach writes about Microsoft research and innovation. Follow him on Twitter.

The post What’s Microsoft’s vision for conversational AI? Computers that understand you appeared first on The AI Blog.

Writing requires a dash of uniquely human creativity. Artificial intelligence alone cannot do it for us, at least not very well. But AI can – and already is – helping us do things like make sure we spell words correctly and use correct grammar, through the myriad ways it is infused across the suite of Microsoft 365 products. Some of them were even used to craft this story.

As the AI in these products is becoming more sophisticated, they are helping us do more than spot a misspelled word.

That includes new intelligent features in Microsoft Word that help us design our documents for maximum readability, along with other features in Microsoft Search and Microsoft Edge that aim to make everybody’s workday more productive. Microsoft showcased these intelligent features today at Microsoft Build, an annual conference for developers, in Seattle.

“Microsoft AI is all about amplifying human ingenuity with intelligent technologies,” said Malavika Rewari, a senior product marketing manager for Microsoft 365.

Microsoft 365 uses AI to help employees overcome some of the realities of modern work, including increasing time demands, overwhelming amounts of data and growing security threats, she noted.

Gathering knowledge

One modern reality of work is age old: a need for knowledge. The difference is that today’s workers turn to the internet to learn, and more than half start with a search engine.

Beginning on May 28, Microsoft Search will move to general availability, the company announced at Build. The technology brings access to the web and work into a single search experience.

Microsoft Search leverages the AI capabilities of Bing and Microsoft Graph, one of the largest collections of data about how people work ever created, enabling workers to find, command, navigate and discover items across their organization’s network of data.

Microsoft Graph includes data from the public internet as well as data available only to employees within an organization such as directories and policy manuals. What’s more, every employee’s graph is distinctive since it contains data that is available only to their specific team, such as documents, and data from their email and calendar.

“We are able to deliver a cohesive search experience that works across any endpoint in Microsoft 365,” said Bill Baer, a senior product marketing manager on the Microsoft Search team. “Whether you are searching in Bing or searching in the Windows 10 search bar, you’ll get a set of contextually relevant results.”

New intelligence in Microsoft Search includes a machine reading comprehension capability that can extract a paragraph from documents explicitly related to your question. For example, if an employee asks, “Can I bring my dog to work?” Microsoft Search will extract the relevant paragraph from the human resources manual and present it as a search result.

“It understands the question you are asking, and then it can find the answer within millions of words of text and give it to you in context,” said Baer.

Another new intelligent feature allows people within a company to conduct people searches with incomplete information. For example, consider being told, “Talk to Pat on the third floor,” and not knowing who Pat is. A search on “Pat, floor 3” uses intelligence from Microsoft Graph such as your immediate team and location to return the most likely Pat, including an office number and picture.

Working on Microsoft Edge

Microsoft, which recently announced plans to adopt the Chromium open source project in the development of Microsoft Edge on the desktop, also is working on ways to make the Edge browser a more natural extension of the Microsoft Search experience, noted Baer. That means users who are signed in to a Microsoft 365 account will be able to see related results within the Edge browser.

The Microsoft Edge team is also experimenting with a feature called Collections that allows users to compile and organize content as they browse the internet in their open browser window and intelligently share the compiled content via email or export it to Excel or Word.

For example, a person shopping for a new camera could visit several product websites and save each page in the Collections pane on the side of the browser. The underlying machine learning in Collections would intelligently display an image of each model along with relevant metadata such as price, user rating and the website where the data originated.

From there, a user could email the list to a friend, or copy and paste the collection elsewhere, maintaining the clean format of the content. Another option is to export to Excel, where the machine learning automatically populates a table organized with columns for brand, model, price, rating and so on based on the collected metadata.

“You can easily, at a glance, get the value and make your decision more quickly,” said Divya Kumar, group product marketing manager for Microsoft Edge. She added that the team is experimenting with similar functionality for exporting to Word, including the ability to compile a document with information such as images and text collected from several websites, citations included.

Better Word documents

Beginning this fall, people working in Word Online who are in search of inspiration and insights on how to make their document better will be able to receive intelligent suggestions with Ideas – a feature that is already making people more productive in PowerPoint and Excel.

The Ideas in Word feature uses machine learning and intelligence from Microsoft Graph to help users write polished prose, create more professional documents and efficiently navigate documents created by others.

For example, feedback and signals from Microsoft Graph indicate that workers generally ignore tools available in Word to structure their documents, such as section heads, but rather manually make some words bold and bigger to indicate a new section.

“Here’s something where we say, ‘Hey, we understand the structure of your document. We can make it navigable, or we could create a table of contents on your behalf,’” explained Kirk Gregersen, a partner director of program management in Microsoft’s Experiences and Devices group.

Other intelligent suggestions include recommended acronyms based on their usage in Microsoft Graph, calculated average time to read the document, highlight extraction, as well as familiar fixes for spelling and grammatical errors and advice on more concise and inclusive language such as “police officer” instead of “policeman.”

Neural rewrites

A recently available intelligent feature in Word is rewrite suggestions, which brings the power of deep learning to offer suggestions on different ways to write a phrase.

The technology builds on enhancements to the popular synonyms feature in Word that use machine learning to understand the context of the sentence the word appears in to offer alternative word choices that are more relevant.

“You don’t need to search online to find an alternative way to express a phrase,” said Zhang Li, a senior program manager in the Microsoft Office team, explaining that the intelligence service will surface suggestions within the document.

His team used similar technology to improve synonym ranking earlier this year, leading the synonym suggestion acceptance rate to double.

“We want to augment your skills,” said Rewari, the senior product marketing manager for Microsoft 365. “We want to help you communicate more efficiently, effectively and inclusively.”

Top video: The Ideas in Word feature uses machine learning and intelligence from Microsoft Graph to help a user style a table for a professional document.

Microsoft Build 2019 – Related links to Microsoft 365 and AI

• Read: New, people centered experiences in Microsoft 365, the world’s productivity cloud
• Check out Microsoft Search and Microsoft Graph
• Read: Microsoft creates AI that can read a document and answer questions about it as well as a person
• Learn more about Microsoft Edge on Chromium
• Check out Word Online and enroll in the Office Insider program for early access to Ideas in Word

John Roach writes about Microsoft research and innovation. Follow him on Twitter.

The post How AI is making people’s workday more productive appeared first on The AI Blog.