Интересный новый метод считывания информации для мозго-машинного интерфейса

vakhnenko — 19.05.2021 В Nature вышла любопытная статья: High-performance brain-to-text communication via handwriting. Русскоязычный научпоп тут: 90 знаков в минуту силой мысли. Есть нечетко сформулированные вещи, конечно, типа "современная электроника вообще не рассчитана на долгую жизнь внутри мозга человека" (а мужики с DBS-то и не знают), но в целом написано вменяемо.

Если вкратце, то исследователи использовали вставленную в моторную кору мозга электродную решетку, попросив испытуемого представить как он от руки на бумаге пишет буквы и предложения. Полученный сигнал разложили на главные компоненты:



(A) Participant T5 attempted to handwrite each character one at a time, following the instructions given on a computer screen (lower panels depict what is shown on the screen, following the timeline). (B) Neural activity in the top 3 principal components (PCs) is shown for three example letters (d, e and m) and 27 repetitions of each letter (“trials”). The color scale was normalized within each panel separately for visualization. (C) Time-warping the neural activity to remove trial-to-trial changes in writing speed reveals consistent patterns of activity unique to each letter. In the inset above C, example time-warping functions are shown for the letter “m”, and lie relatively close to the identity line (each trial’s warping function is plotted with a differently colored line). (D) Decoded pen trajectories are shown for all 31 tested characters: 26 lower-case letters, commas, apostrophes, question marks, tildes (~) and greater-than signs (>). Intended 2D pen tip velocity was linearly decoded from the neural activity using cross-validation (each character was held out). The decoded velocity was then averaged across trials and integrated to compute the pen trajectory (orange circles denote the start of the trajectory). (E) A 2-dimensional visualization of the neural activity made using t-SNE. Each circle is a single trial (27 trials for each of 31 characters).

...и совместили это с глубоким обучением. В результате скорость печати смогла выйти на "плато" в 90 знаков в минуту, правда при таком темпе печати кол-во ошибок большое, но это лечится добавкой языковой модели:



(A) Diagram of our decoding algorithm. First, the neural activity (multiunit threshold crossings) is temporally binned (20 ms bins) and smoothed on each electrode. Then, a recurrent neural network (RNN) converts this neural population time series (xt) into a probability time series (pt-d) describing the likelihood of each character and the probability of any new character beginning. The RNN has a one second output delay (d) so that it has time to observe the full character before deciding its identity. Finally, the character probabilities were thresholded to produce “Raw Output” for real-time use (when the “new character” probability crossed a threshold at time t, the most likely character at time t+0.3 was emitted). In an offline retrospective analysis, the character probabilities were combined with a large-vocabulary language model to decode the most likely text that the participant wrote (we used a custom 50,000-word bigram model). (B) Two realtime example trials are shown, demonstrating the RNN’s ability to decode readily understandable text on sentences it was never trained on. Errors are highlighted in red and spaces are denoted with “>“. (C) Error rates (edit distances) and typing speeds are shown for five days, with four blocks of 7-10 sentences each (each block indicated with a single circle). The speed is more than double that of the next fastest intracortical BCI7.

Разумеется, в таком виде это всё ещё нужно только инвалидам. У среднего человека скорость печати более чем в два раза выше, ~200 знаков в минуту. Но всё равно интересная идея, хоть что-то новое с первой половины нулевых, а то все ограничивались двиганьем курсора.

Оставить комментарий

Популярные посты:

• Ранее
• Архив