cognitive science
and more
How to write with your mind? (or: Decoding attention through pupillometry)

I'm very excited to announce that our article The Mind-Writing Pupil is now available at PLoS ONE! In this article, we describe a technique that allows you to—literally—write by thinking of letters. This sounds like a wild claim, but it's not; the technique is actually surprisingly simple. In the video below you can see a demonstration:

So how does this work?


First, let's look at the simple case in which there are only two letters: A and B. Both letters are presented on backgrounds that alternate between bright and dark; crucially, whenever the A is dark, the B is bright, and vice versa.

Now imagine that you are looking directly at the B. In this case, your eye's pupil would respond strongly to the brightness of the B (or rather its background); this is the pupillary light response: the constriction (shrinkage) of the pupil in response to brightness, and dilation (expansion) in response to darkness.

So if we would measure the size of your pupil while you're looking directly at the B, we would see that it follows the B's brightness. In this example, the B is initially bright, so the pupil would be small; then the B turns dark, so the pupil would become large; and then the B turns bright again, so the pupil would become small again. Therefore, we could simply measure how pupil size changes over time, and use this to determine with almost 100% certainty which of the two letters you're looking at. This is easy, and, if fact, our technique can be used this way as well. However, this isn't really writing your mind, because you move your eyes toward one of the letters; so we're really just measuring eye movements, albeit indirectly.

But—and this is were things get interesting—you don't need to look directly at something bright in order for your pupil to constrict: A covert shift of attention is enough to trigger a (weak) pupillary light response. Phrased differently, if you attend to something bright from the corner of your eye, without actually looking at it, your pupil will still constrict. (I've written about this before.)

In our technique, this means that participants do not need to move their eyes at all. They can simply keep their eyes in the center of the screen, and covertly shift their attention to the letter that they want to select. This will have the same effect (but weaker) on the size of the pupil as looking directly at the letter, and we can therefore figure out which letter the participant is attending to using the same logic as described above: by measuring how pupil size changes over time.

So that's all there is to it: We measure the size of the pupil, and determine whether it changes in a rhythm that corresponds to the letter A (large→small→large) or B (small→large→small). It takes some time to do this, because we need several dark-bright cycles to make a reliable decision, but it works remarkably well. And a simple extension of this technique, which we describe in the paper (and is shown in the video above), allows you to write sentences using the entire alphabet. To give some examples, one participant in our experiment wrote enfin terminée (finally finished), indicating that he/she was completely fed up with the very long experiment that we used to test our technique! Another participant wrote je ne suis pas si rapide que ça (I'm not all that fast), which (as far as experimental results go) I thought was funny, because he/she was by far the fastest writer of all participants that we tested.

So the technique works: You can write by thinking of letters, or, more specifically, by covertly attending to them.

But what's the point? What can you do with this?

Well, first, as a fundamental researcher, I just think it's fascinating that this is even possible, and I believe that it tells us something about the relationship between attention, eye movements, and pupil size. But our technique may also have practical applications as a human-computer interface for people that suffer from complete paralysis.

When someone is completely paralyzed and unable to communicate, yet remains conscious, he or she is in a locked-in state. This happens most commonly as the final stage of ALS (amyothropic lateral sclerosis), but it can result from other causes as well, notably brain injuries. Many techniques have been developed that provide a communication channel for locked-in patients; these techniques generally rely on measurement of brain activity. For example, by presenting flashing letters on a screen and simultaneously measuring brain waves, you can find out which of the letters someone is attending to. This is similar in concept to, but technically very different from, our pupillometric technique.

The downside of recording brain activity is that it is difficult; it requires expensive equipment, careful preparation (e.g. to apply electrodes to the skull), careful calibration, etc. Some of these practical problems would be solved, or at least reduced, if we could rely on pupil size instead of brain activity.

However, there are a lot of ifs here. First, we have only tested our technique in healthy participants, and it remains to be seen whether it will also work in paralyzed patients; for example, it may be too difficult, or these patients may not have an intact pupillary light response. So there is plenty of work to be done. But, for the moment, I'm simply excited to present this prototype—but fully functional—system. And it is completely free and open for anyone to use and improve.

Mathôt, S., Melmi, J.-B, van der Linden, L., & Van der Stigchel. (2016). The mind-writing pupil: A human-computer interface based on decoding of attention through pupillometry. PLoS ONE. http://doi.org/10.1371/journal.pone.0148805