Skip to main content

Blog

Learn About Our Meetup

5000+ Members

MEETUPS

LEARN, CONNECT, SHARE

Join our meetup, learn, connect, share, and get to know your Toronto AI community. 

JOB POSTINGS

INDEED POSTINGS

Browse through the latest deep learning, ai, machine learning postings from Indeed for the GTA.

CONTACT

CONNECT WITH US

Are you looking to sponsor space, be a speaker, or volunteer, feel free to give us a shout.

[R] Speech synthesis from neural decoding of spoken sentences (Nature)

Speech synthesis from neural decoding of spoken sentences

Abstract

Technology that translates neural activity into speech would be transformative for people who are unable to communicate as a result of neurological impairments. Decoding speech from neural activity is challenging because speaking requires very precise and rapid multi-dimensional control of vocal tract articulators. Here we designed a neural decoder that explicitly leverages kinematic and sound representations encoded in human cortical activity to synthesize audible speech. Recurrent neural networks first decoded directly recorded cortical activity into representations of articulatory movement, and then transformed these representations into speech acoustics. In closed vocabulary tests, listeners could readily identify and transcribe speech synthesized from cortical activity. Intermediate articulatory dynamics enhanced performance even with limited data. Decoded articulatory representations were highly conserved across speakers, enabling a component of the decoder to be transferrable across participants. Furthermore, the decoder could synthesize speech when a participant silently mimed sentences. These findings advance the clinical viability of using speech neuroprosthetic technology to restore spoken communication.

Nature (paywall): https://www.nature.com/articles/s41586-019-1119-1

direct link to pdf: https://www.gwern.net/docs/ai/2019-anumanchipalli.pdf

YouTube demo: https://www.youtube.com/watch?v=kbX9FLJ6WKw

Articles about this paper:

Nature: https://www.nature.com/articles/d41586-019-01328-x

Scientific American: https://www.scientificamerican.com/article/scientists-take-a-step-toward-decoding-thoughts/

NY Times: https://www.nytimes.com/2019/04/24/health/artificial-speech-brain-injury.html

submitted by /u/sensetime
[link] [comments]