How to integrate word prediction keyboard into P300-based BCI (brain computer interface) system?

Question

See this question in Cognitive Sciences.

A P300 based BCI is designed for locked-in patients to communicate, P300 is a brain wave elicited in the process of decision making. Farwell and Donchin (1988) described a system with a 6x6 speller matrix, and various columns or rows are highlighted. "When a column or row contains the character a subject desires to communicate, the P300 response is elicited (since this character is “special” it is the target stimulus described in the typical oddball paradigm). The combination of the row and column which evoked the response locates the desired character."(see an example in Fig 1.). As commented by Rumi P., you can see this system as a black box, but flashing display is required to trigger a stimulus(P300) in the user (comment from Jayfang) because the target users are locked-in patients. The purpose to add word prediction is to increase its speed, currently, patients can accurately spell 3.4–4.3 chars/min.

According to this system, only one target can be detected each time, thus only one flashing group (rows and columns) on the screen. If we use word prediction keyboard in this system, once a letter is selected, the predicted words will appear on the screen. If the predicted words flash together with the rows and columns, the system is not able to tell which (words or letters) to select. What's the best way to integrate word prediction keyboard into this system? two ideas below:

1. to have the keyboard disappeared after a letter is selected, only leave predicted words appear on the screen. If nothing is selected in a limited time, remove the words and the keyboard comes back, subject continues to spell.
2. same idea as above, instead of removing the keyboard, just to stop the keyboard from flashing when the predicted words are flashing after a letter is selected. However, both might slow down the spelling process if subjects only need words with 3 or fewer letters. Any other ideas?

Fig. 1. An example from the Brain-Computer Interface Laboratory at East Tennessee State University [To spell the word "DOG", as rows and columns flash successively, the user has to count how many times the letter 'D' (the target) flashes. This results in a P300 response being generated each time the row or column containing the target flashes. The twelve-flash series is repeated a predetermined number of times. The responses for each row and column are averaged, and a classifier is applied to determine how closely each averaged response resembles the P300. The intersection of the row and column with the highest classification values is selected. In this case, the row and column containing the target letter 'D' would be selected, and a "D" would be presented as feedback to the user on the line below the presented word "DOG" at the top of the matrix.]

Farwell, L. A., & Donchin, E. (1988). Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalography and clinical Neurophysiology, 70(6), 510-523. doi:10.1016/0013-4694(88)90149-6

Answer 1

Not a definite answer, because I have never worked with this kind of device. But it's probably a suggestion worth pursuing if you're in this area.

The way I understand it, the system knows what was recognized because of the time at which it was recognized. The user who wants to spell "D" sees "D" flash, the system notices his brain generated a P300 at the time D flashed, the system concludes that the user wanted to spell D. You cannot put more than one "wanted" stimulus on the screen at the same time, because you can't know which of them triggered the P300. So you are stuck with showing the stimuli in series and wait for a reaction.

As a UX specialist, my first idea is to combine the system with a gaze recording device. These are quite common technology nowadays, and their price will probably be negligible when compared to your EEG setup. Their temporal resolution is good enough that they can record individual saccades.

A decision does have some delay neurologically, but it's on the scale of a few hundred milliseconds, and I think the variation is not so terribly high. If it is known where the patient was gazing at during the last few milliseconds, it should be possible to restrict the possible gaze fixations which triggered the decision to a very small number. Thus you should be able to show multiple stimuli on the same screen and distinguish between them (again based on the time at which they were seen) as a trigger of the decision. I think the reason the original authors never thought of it is simply that gaze recognition technology was very primitive in the eighties. If they knew of it, it probably did not offer sufficient temporal and spatial resolution.

Please consult a neurologist about the feasibility of this approach. My knowledge of neuroscience is restricted to more basic material, and there might be an obvious problem I'm overlooking. I hope that, when working on such a project, you have access to a domain expert.

Answer 2

Have you considered simply increasing the amount of letters entered before displaying or offering the predictive text? By setting a gate at – say – three letters you avoid the short word problem, and also greatly trim the list of possible responses for your predictive selections.

After which, you could follow your own suggestion of flashing the predictive text first, then moving across the content in the normal pattern.