4

See this question in Cognitive Sciences.

A P300 based BCI is designed for locked-in patients to communicate, P300 is a brain wave elicited in the process of decision making. Farwell and Donchin (1988) described a system with a 6x6 speller matrix, and various columns or rows are highlighted. "When a column or row contains the character a subject desires to communicate, the P300 response is elicited (since this character is “special” it is the target stimulus described in the typical oddball paradigm). The combination of the row and column which evoked the response locates the desired character."(see an example in Fig 1.). As commented by Rumi P., you can see this system as a black box, but flashing display is required to trigger a stimulus(P300) in the user (comment from Jayfang) because the target users are locked-in patients. The purpose to add word prediction is to increase its speed, currently, patients can accurately spell 3.4–4.3 chars/min.

According to this system, only one target can be detected each time, thus only one flashing group (rows and columns) on the screen. If we use word prediction keyboard in this system, once a letter is selected, the predicted words will appear on the screen. If the predicted words flash together with the rows and columns, the system is not able to tell which (words or letters) to select. What's the best way to integrate word prediction keyboard into this system? two ideas below:

  1. to have the keyboard disappeared after a letter is selected, only leave predicted words appear on the screen. If nothing is selected in a limited time, remove the words and the keyboard comes back, subject continues to spell.
  2. same idea as above, instead of removing the keyboard, just to stop the keyboard from flashing when the predicted words are flashing after a letter is selected. However, both might slow down the spelling process if subjects only need words with 3 or fewer letters. Any other ideas?

Fig. 1. An example from the Brain-Computer Interface Laboratory at East Tennessee State University [To spell the word "DOG", as rows and columns flash successively, the user has to count how many times the letter 'D' (the target) flashes. This results in a P300 response being generated each time the row or column containing the target flashes. The twelve-flash series is repeated a predetermined number of times. The responses for each row and column are averaged, and a classifier is applied to determine how closely each averaged response resembles the P300. The intersection of the row and column with the highest classification values is selected. In this case, the row and column containing the target letter 'D' would be selected, and a "D" would be presented as feedback to the user on the line below the presented word "DOG" at the top of the matrix.]

Farwell, L. A., & Donchin, E. (1988). Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalography and clinical Neurophysiology, 70(6), 510-523. doi:10.1016/0013-4694(88)90149-6

  • This question appears to be off-topic because it is about integration and implementation. Seems more of a fit on StackOverflow. I don't see the UX question here. – Code Maverick Jan 15 '15 at 2:50
  • 2
    Welcome to the site, @Sophy! I think this question is on-topic because you appear to be asking about which interaction pattern will best fulfill user needs, but I'm having a bit of trouble parsing your post because the language is so esoteric. Can you please edit your post to clarify the following: (1) what is a P300? (2) what is the purpose of a BCI system? (3) who is the target user group? (4) what does the user hope to accomplish using the system? Why? (5) are there any significant constraints? (6) what are you optimizing for? – Graham Herrli Jan 15 '15 at 4:58
  • 2
    Also, please note that if you're going to cross-post between here and CognitiveSciences.SE, it's generally a good idea to ask slightly different questions in each place to cater to the expertise of the people on each site. – Graham Herrli Jan 15 '15 at 4:59
  • 3
    Hi Sophy, we will have a difficult time giving a proper answer given a BCI's uniqueness - unless someone has direct experience with such a system. Your best solution is likely to implement both and do user testing. – Evil Closet Monkey Jan 15 '15 at 5:10
  • Is a flashing display required to trigger a stimulus in the user, or is it merely an "active row" indicator? (1988 device screens were quite basic) – Jason A. Jan 15 '15 at 9:31
2

Not a definite answer, because I have never worked with this kind of device. But it's probably a suggestion worth pursuing if you're in this area.

The way I understand it, the system knows what was recognized because of the time at which it was recognized. The user who wants to spell "D" sees "D" flash, the system notices his brain generated a P300 at the time D flashed, the system concludes that the user wanted to spell D. You cannot put more than one "wanted" stimulus on the screen at the same time, because you can't know which of them triggered the P300. So you are stuck with showing the stimuli in series and wait for a reaction.

As a UX specialist, my first idea is to combine the system with a gaze recording device. These are quite common technology nowadays, and their price will probably be negligible when compared to your EEG setup. Their temporal resolution is good enough that they can record individual saccades.

A decision does have some delay neurologically, but it's on the scale of a few hundred milliseconds, and I think the variation is not so terribly high. If it is known where the patient was gazing at during the last few milliseconds, it should be possible to restrict the possible gaze fixations which triggered the decision to a very small number. Thus you should be able to show multiple stimuli on the same screen and distinguish between them (again based on the time at which they were seen) as a trigger of the decision. I think the reason the original authors never thought of it is simply that gaze recognition technology was very primitive in the eighties. If they knew of it, it probably did not offer sufficient temporal and spatial resolution.

Please consult a neurologist about the feasibility of this approach. My knowledge of neuroscience is restricted to more basic material, and there might be an obvious problem I'm overlooking. I hope that, when working on such a project, you have access to a domain expert.

| improve this answer | |
  • Thank you for the answer. You understand it right about the conflicts on selection, but it's not because of time, but because of the brain wave (P300). Since the function of flashing is equaled to be highlighted, will be selected once the brain wave is detected. If there are two groups of flashing (word and letter) at the same time, when the brain wave is detected, the system won't know which to select from. But if have them flash separately, it will be a waste of time if the predicted words have to flash each time after a letter is selected. – Sophy Jan 15 '15 at 22:48
  • @sophy Maybe I didn't understand all. Is your recognition rate limited by a) the time needed for the brain creating a P300 and you recording it, or b) the speed with which you flash screens? If b), then you can flash two things at the same screen, say a letter in the lower right corner and a suggested word in the upper left corner. An eye tracking device can then tell you the exact point on the screen the patient was looking at when the P300 appeared (and the history before it, recording discrete eye movements). If it is a), then nothing you can do to the screen will give you (cont.) – Rumi P. Jan 17 '15 at 14:20
  • (cont.) an improvement in speed. In this case, I don't understand what kind of answer you are looking for - you are already limited by the speed of the human brain and not of the interface, so no solution can be quicker than showing the letters and then the suggested word after each other. – Rumi P. Jan 17 '15 at 14:21
  • The speed is limited by a) the time needed for the brain creating a P300 and the displaying letters. We think to add suggested word will improve the spelling speed. However, an eye tracking device would not work for patients losing eye movements, thus we are trying to limit the words and letters in one screen, also separate their flashing, so the system can tell which one elicits the brain wave. – Sophy Jan 17 '15 at 16:48
2

Have you considered simply increasing the amount of letters entered before displaying or offering the predictive text? By setting a gate at – say – three letters you avoid the short word problem, and also greatly trim the list of possible responses for your predictive selections.

After which, you could follow your own suggestion of flashing the predictive text first, then moving across the content in the normal pattern.

| improve this answer | |
  • Good idea! Might not be an easy job for me to code, but I will try. Thank you! – Sophy Jan 17 '15 at 1:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.