New breakthrough in AI+ brain-computer interface!

"The inability to have a normal conversation and express myself in any way is devastating
.

I had hoped that I never woke up from a coma
.

" Pancho wrote, "This research has changed my life, as if it gave me the first prize.
A second chance to speak
.

" Pancho (pseudonym) expressed the above feelings in an interview with The New York Times
.

Through AI and brain-computer interface technology, the ambiguous voice of Pancho, who has lost the ability to speak for 18 years, is accurately analyzed
.

 He is a patient in a study published in the New England Journal of Medicine (NEJM) this Thursday
.

This epoch-making technology involves multiple fields, so the research team not only includes neurosurgeons, neuroscientists, and linguists, but also brings together engineers and computer scientists.
It is a veritable medical-industrial team
.

Pay attention to the August 29 "AI+Medical: Innovation, Smart Future" summit attended by the editor-in-chief of the New England Journal of Medicine, and discuss the cutting-edge progress of medical AI with industry leaders
.

See the poster at the end of the article for details
.

In this study, Edward F.
Chang, a neurosurgeon at the University of California, San Francisco (UCSF), and his team implanted electrode arrays into the cerebral cortex of patients with dysarthria after stroke.
The neural signals collected by the electrode arrays passed through AI The technology was converted into words, deciphering the patient’s ambiguous language at a rate of 15.
2 words per minute and an error rate of 25.
6%
.

Dysargia is a common sequelae of stroke
.

Such patients still have a normal mind, but are unable to pronounce or make comprehensible speech due to vocal-related muscle control disorders after a stroke
.

The decoding speed of 15 words per minute is far lower than the normal speaking speed
.

But in the previous ten years, Pancho could only make grunting and groaning sounds, and because of the stroke caused severe spastic quadriplegia, he could only wear a baseball cap and touch the touch screen with a pointer attached to the cap every minute.
Type and communicate at a speed of 5 words
.

"NEJM Frontiers in Medicine" commented on July 15 also pointed out that although the accuracy of words and sentences is not high, "the input speed of patients has been doubled compared to before
.
"
More importantly, the paper reveals a new research direction, that is, "decoding entire words directly from the brain region that controls language"
.

Collecting signals The first step in analyzing brain language signals is to accurately collect the appropriate signals from the appropriate parts of the brain
.

In February 2019, Chang's team performed a craniotomy for Pancho, which lasted about 3 hours, placing a high-density electrocortical electrode array on the surface of the pia mater in the subdural space
.

The electrode array is rectangular (6.
7 cm×3.
5 cm×0.
51 mm), composed of 128 flat disc-shaped electrodes, the electrodes are arranged in a 16×8 lattice structure, and the distance between the centers of two adjacent electrodes is 4 mm
.

Figure 1.
The display screen presents a paragraph to Pancho, and the patient will try to answer using the words in the vocabulary (containing 50 words)
.

At the same time, cortical signals are collected from the surface of the brain by the electrode array and processed in real time
.

Figure 2.
The patient's brain reconstruction map and the position of the implanted electrodes, and the role of the electrodes on language detection and word classification models
.

The drawn electrode size (area) and opacity are proportional to the relative effect (important electrodes are drawn larger and more opaque)
.

The center front back is highlighted in light green
.

Electrodes cover multiple cortical areas involved in language processing, and Pancho's cerebral cortex electrograms are collected
.

The other end of the electrode array is connected to a percutaneous connector fixed outside the convex face of the contralateral skull, and the cortical signal from the electrode array is transmitted to the computer for real-time signal analysis
.

According to previous research, neural activity in the 70-150 Hz frequency range called high-frequency gamma wave is related to language processing.
Therefore, Chang's team measured the activity in the high-frequency gamma band of each channel for subsequent analysis.
And real-time decoding
.

The electrocorticogram collected from the decoded language needs to undergo complex transformation before it can be parsed into language
.

In order to create a suitable analytical method, in this study of 50 tests lasting 81 weeks, Chang's team designed two tasks for Pancho and collected neural activity signals when he completed the tasks
.

Pancho’s first task is to try to say 50 commonly used independent English words, which mainly include words related to nursing or patient needs, such as "hungry" (hunger), "music" (music) and "computer" (computer) )
.

His second task is to try to say 50 English sentences composed of these words as quickly as he feels comfortable (in order), such as "My nurse is right outside", "Bring my glasses, please" (please pass me glasses) and so on
.

Researchers use neural activity data collected during testing to train, fine-tune, and evaluate custom models
.

They created models for language detection and word classification, and the models use deep learning techniques to make predictions based on neural activity
.

In order to decode sentences based on Pancho's neural activity in the sentence task in real time, Chang et al.
also used a natural language model and a Viterbi decoder
.

Figure 3.
The processed neural signals are analyzed one by one by the language detection model to detect Pancho's attempts
.

The classifier calculates the probability of a word (out of 50 possible words) from each window of relevant neural activity detected
.

The Viterbi decoding algorithm uses these probabilities and word sequence probabilities to decode the most likely sentence based on neural activity data
.

The predicted sentence (updated every time a word is decoded) is displayed to Pancho as feedback
.

The language detection model processes each time point of neural activity in the task and detects Pancho's attempts to speak words in real time
.

The neural activity data and time collected in the word task are used to fit the model
.

For each detected attempt, the word classification model processes the neural activity from 1 second before the detection of the attempted speech to 3 seconds after the beginning of the speech to predict the probability of the occurrence of the word, and the predicted probability of the occurrence of the word can quantify Pancho's attempt to speak this Possibility of words
.

The research team also created a natural language model based on the characteristics of English, which can derive the probability of the next word based on the words in front of the sentence
.

Figure 4.
The number of word errors in decoded sentences with or without the natural language model
.

Each small vertical line represents the number of word errors in one attempt (3 attempts for each target sentence)
.

Each point represents the average number of errors in the target sentence in 3 attempts
.

The histogram shows the total number of errors of 150 attempts
.

 Figure 5.
Taking 7 target sentences as an example, the decoding situation with and without natural language model
.

Black: Decode the word correctly
.

Red: Wrongly decoded words
.

The last part of the decoding method uses a Viterbi decoder, which can determine the most likely word sequence based on the word probability predicted by the word classifier and the word sequence probability predicted by the natural language model
.

By incorporating the language model, the Viterbi decoder can decode seemingly more reasonable sentences, rather than just string together the words predicted by the word classifier
.

A huge challenge Because the recorded electrocortical signals are relatively stable, researchers can accumulate large amounts of data through long-term recording of cortical activity, train decoding models, and improve decoding accuracy and speed
.

Chang said that in the future, the team also hopes to design an electrode array with higher sensitivity and make it wireless to achieve complete implantation and avoid infection
.

As more and more patients participate, there may be language decoding problems caused by individual brain differences
.

In addition, when patients feel tired or sick, their electrocorticogram signal strength will also change
.

These are challenges that need to be addressed
.

Copyright information This article was translated, written or commissioned by the "NEJM Frontiers of Medicine" jointly created by the Jiahui Medical Research and Education Group (J-Med) and the "New England Journal of Medicine" (NEJM)
.

The Chinese translation of the full text and the included diagrams are exclusively authorized by the NEJM Group
.

If you need to reprint, please leave a message or contact nejmqianyan@nejmqianyan.
cn
.

Unauthorized translation is an infringement, and the copyright owner reserves the right to pursue legal liabilities
.