Nature | A New Breakthrough in Brain-Computer Interaction: Thoughts and Words

Author | Edited by Su Yiting | Xi If Stephen Hawking can live to this day, May 12, 2021 may be a particularly exciting day for him.

From the earliest spelling card to typing with one finger, until the condition deteriorated to the point where only one muscle of the cheek could be used to control the computer, two characters were completed in one minute.

He will definitely feel frustrated and helpless, but in any case, the top human-computer interaction technology was specially developed for him.
This is the era he has gone through.

But he is unfortunate.
If he can come earlier on May 12, 2021, the scientific giant who has been suffering from ALS for more than 50 years may be able to continue to show the highest level of human thinking in a more decent way.
.

What exactly happened on May 12, 2021? On this day, Krishna Sheonoy's team from Stanford University School of Medicine published an article High-performance brain-to-text communication via handwriting in Nature.
They recorded the neural activity of the motor cortex of paralyzed patients, combined with neural network analysis, and successfully decoded in real time.
Out of the words and sentences written by the patient with his mind.

This brain-computer interaction technology can reach a typing speed of 90 characters per minute, which is equivalent to the typing speed of a normal person playing on a mobile phone, and the accuracy rate after offline automatic correction is as high as 99%.
This is a milestone for mankind.

Sure enough, after the publication of this study, there was an upsurge of discussion in the scientific community.

This subject was code-named T5.
He was completely paralyzed from the neck down due to spinal cord damage, and his hands were completely non-functional.

The researchers implanted two microelectrode arrays in the hand motor brain area of ​​T5 to record nerve signals.

After recovery from the operation, they asked T5 to imagine that they could regain the feeling of writing on paper with a pen in both hands.
In order to make the nerve signals easier to be detected by the machine, they agreed that spaces should be represented by a greater than sign ('>'), and the period should be represented by a tilde.
('~') means that other characters remain unchanged.

Although paralyzed for many years, the researchers are delighted to find that T5 has not forgotten how to write.
They detected that the neural signals of these writings are very strong, and the neural activity is very similar every time they write the same character.

If the neural signal is divided into different neural activity clusters by nonlinear dimensionality reduction method, these characters can be distinguished well, and the accuracy rate is as high as 94.
1%.

This means that T5 may be able to write words or even sentences with his mind.

Next, the researchers trained a recurrent neural network (RNN) to decode the information that T5 wanted to convey.

They encountered two difficulties: First, they could not know the time for writing each character in T5 in advance, so it was difficult to use supervised learning methods to calculate, and secondly, the amount of data they collected was incomparable with the amount of normal RNN data, and it was inevitable.
Fitting problems (referring to matching a specific data set too accurately to fit other data well or predict future observations).

So they decided to try the data from the previous few days to do a preliminary RNN analysis, and then add the collected data little by little to make corrections to continuously improve the neural network.

On the last day of the test, they finally completed 7.
6 hours, 31574 characters of data, at this time the RNN network database finally began to take shape.

In order to test the performance of the RNN algorithm, T5 wrote a lot of sentences with mind.
Some of these sentences are required to be written, and some are free to use when answering open questions.

What’s shocking is that, in either case, the RNN neural network can display highly readable words on the computer screen with a delay of about half a second.
The error rate is only 5.
4%, and if you use offline After the language model is automatically calibrated, the error rate is reduced to 0.
89%.
If all the data is finally retrained to form a new RNN, the error rate is even only 0.
17%, which means that T5 can completely "think into words at this time" "Again.

It’s just that if applied to the clinic, this brain-computer interaction method still has a real problem that needs to be solved: in this study, the brain’s neural signals will also change with the passage of time, so T5 needs to imagine how many times each day.
Ten sentences provide correction information for the machine, which undoubtedly puts a great burden on the patient.

Fortunately, the researchers found that if every 2-7 days T5 can imagine 10 sentences (instead of the previous 30 sentences) as calibration information for RNN analysis, this network can also achieve unsatisfactory results (error rate 8.
5%) , And if a new automatic calibration method for unsupervised learning language models is adopted, the error rate of mind calibration without T5 can be as low as 7.
3% (offline calculation can be as low as 0.
84%)-this is undoubtedly acceptable Scope out.

Why is the decoding speed in this research so fast? This may be related to the high variability of written characters in the space and time dimensions.

Compared with simply drawing a straight line from the starting point to the end point, each character written in T5 has its own unique strokes (space dimension) and speed (time dimension), so the corresponding brain nerve activity is also more different, so it is easier to convert Its separate.

Researchers believe that compared to the space dimension, the time dimension is the main reason for the RNN to increase the character discrimination.

For example, the previous brain-computer interface technology often relied on eye movement information, mainly outputting characters through the eyes staying on the position of the keyboard screen.
This point-to-point linear motion is more difficult to decode than the neural signals generated by handwritten letters, so it also explains why The reason why eye movement decoding can only reach 45 characters per minute.

This landmark study is undoubtedly a powerful cardiotonic for paralyzed patients.
Of course, there are still many problems that need to be solved in the future, such as how to ensure that "one implantation is effective for life"? How to promote from English to other languages? This research is just a wonderful beginning.
The authors have made their research data fully publicized and encouraged the cooperation of all fields of science.
I believe that more and more scientists will stand on their shoulders in the future and go all out to overcome these problems.
.

Original link: https:// Platemaker: Instructions for reprinting on the eleventh [Original article] BioArt original article, personal forwarding and sharing are welcome, reprinting without permission is prohibited, published The copyright of all works of is owned by BioArt.

BioArt reserves all statutory rights and offenders must be investigated.