NASA Research: Reading Thoughts Using Electromyography

Victims of mind control report having their thoughts read especially those that are verbalized internally otherwise know as subvocal speech. Subvocal speech happens also when we read although some “speed reading” techniques try to get you to stop doing this because it slows the reading process down. It has long been known that during subvocal speech electrical signals are still sent to those muscles of the face and throat that participate in ordinary speech even though no sounds are generated. These signals can be detected in the form of electromyograms (EMG) using electrodes placed in the face and neck area.

In the previous article we saw how Lawrence Pinneo found some success in using electromyograms and electroencephalograms to detect thoughts in the form of subvocal speech. That was thirty years ago, this article looks at current day attempts to do this through the work of Charles Jorgenson at NASA. Jorgensen works at NASA’s Ames Research Center where  they have been investigating alternative methods of communication and control in hostile environments where normal methods are not always possible. Examples include astronauts, fighter pilots and rescue workers.

In contrast to Pinneo, Jorgensen placed electrodes in the neck region only. In the first stage of processing signals from the electrodes were sampled at up to 10,000 times per second and run through a 60 HZ notch filter to exclude line interference and band pass filters to remove anything outside the 30 – 500 HZ range.

The next stage in processing marks a major change from Pinneo’s work in which the data is transformed from the time domain to the frequency domain. Jorgensen experimented with a number of transforms including Fourier and Wavelet transforms to do this and seemed to settle on a quad tree wavelet transform.

During the final phase features extracted from the transforms were input into different machine learning algorithms to train the system to pattern match the words or phonemes being examined. Jorgensen experimented with a number of techniques including neural networks and support vector machines. Once the system had been trained it was then used to attempt to match against new signals. In a range of different applications was able to achieve around 74% success for small vocabulary sets of up to 15 words.

The results continue to prove the feasibility of thought reading using subvocal recognition. However given the vast increase in computing power plus the many advancemnets in signal processing and pattern recognition since Pinneo’s time these results are somewhat disappointing although I should point out that nothing has been published by Jorgensen in 2-3 years. Not sure if we  should read anything into that.

Despite this, some companies are looking to commercialize this technology. For example NTT DoCoMo are working on a subvocal mobile phone with the idea that people can answer their phone without annoying those around them such as in a movie theater. NASA are also working with QUASAR corporation to develop better sensors. In April 2006 Forbes magazine published an article on Jorgensen’s work entitled The Silent Speaker:

Jorgensen sees the day when electromagnetic sensors will be woven into the fibers of turtlenecks or rescue workers’ outfits. “As long as people have had machines and tools, they’ve been dependent on the physicality of the body,” Jorgensen says. “Separate those control activities from the body and it opens a whole new generation of interface design.”

QUASAR recently announced that it is working with NASA Ames Research Center to develop a hands-free UGV (unmanned ground vehicle) control system based on subvocal speech and forearm EMG. The purpose of the system is to allow soldiers to control the devices without having to set down their weapons or other equipment.

In December 2001 NASA’s Ames Research Center made a presentation to NorthWest Airlines in which they stated that they were working with a commercial partner to develop neuro-electric sensors to remotely monitor the EEG and ECG of passengers at airport security. This information was leaked to the Washington Times which published an article about it. The article drew denials from NASA and the Washington Times no longer has the article up on their site. The Electronic Privacy Information Center obtained details of the presentation under the Freedom of Information Act which can be seen here.

References

1. Techniques of EMG signal analysis: detection, processing, classification and applications, Raez, Hussain and Mohd-Yasin 2006
2. Small Vocabulary Recognition Using Surface Electromyography in an Acoustically Harsh Environment,  Bradley J. Betts, Charles Jorgensen, 2005
3. Web Browser Control Using EMG Based Sub Vocal Speech Recognition, Chuck Jorgensen, Kim Binsted, 2004
4. NASA Develops System To Computerize Silent, “Subvocal Speech”, NASA News,
5. The Silent Speaker, David Armstrong, Forbes Magazine, April 10 2006

Early Research – Lawrence Pinneo

The work of Lawrence Pinneo in the early seventies often comes up in mind control discussions. It is cited as evidence that a thought reading capability has been around for some time. A report by Pinneo is available at the “Christians Against Mental Slavery” website, so I decided to download it and take a look at what it was all about.

The report, “Feasibility Study for Design of a Biocybernetic Communication System”, written in 1975  details the findings of a three year study sponsored by DARPA and conducted by a team from SRI (Stanford Research Institute) lead by Pinneo, a neurophysiologist and electronic engineer. The work was part of a larger DARPA Biocybernetics Program whose directive was to evaluate the potential of biological measurable signals helped by real-time computer processing, to assist in the control of vehicles, weaponry, or other systems.

The work is remarkable for the time given the computing power available. The computers used were a CDC 6400 a machine and a LINC-8, machines which were typically loaded using punched cards and paper tape. The laptop I’m writing this blog on is probably on the order of many thousand times more powerful than these machines.

The stated goal of the research was

… to test the feasibility of designing a close-coupled, two-way communication link between man and computer using biological information from muscles of the vocal apparatus and the electrical activity of the brain during overt and covert (verbal thinking) speech. The research plan was predicated on existing evidence that verbal ideas or thoughts are subvocally represented in the muscles of the vocal apparatus.

Covert speech is more commonly called subvocal speech today and refers to verbal thinking where you think to yourself in words. At the time it was known that even though no sounds are generated during this covert speech electrical signals are still sent to the muscles of the face and throat which can be detected using electromyography similar to an electroencephalography. So the goal was essentially to investigate electrical signals from the brain (EEG) and muscle nerves (EMG) during covert and overt speech and to determine what information could be extracted from them.

Electrodes were placed on both the head and facial musculature  to capture EEG and EMG readings. See the original document for exact placement. Subjects were placed in a shielded booth where words were presented on a screen and they had to either say(overtly) or think (covertly) the word they saw. The signals from the electrodes were captured and fed through an analog to digital converter and into the computer.

The processing of the signals was fairly straightforward. For the study he worked with a set of 15 words.For each word the signal amplitude was sampled over a 6 second period – three seconds before to three seconds after the word is uttered or thought. 255 samples were taken in total. For each word he constructed a template, basically all 255 sample points averaged over ten repetitions of the word.

When it came to predicting which word a subject was thinking of he would simply compare the measured sample to each of the 15 templates and calculate the root mean square(RMS) of each templates difference from the measured value. The word whose template had the lowest RMS value was chosen as the word the subject was deemed to be thinking.

This type of processing was repeated for a number of subjects for both overt and covert speech. A wide range of experimental factors was also tested such as electrode placement, male versus female, etc.

The report has a number of findings, here are some of the highlights

• EEG responses for covert speech mimicked those of overt speech for the same subject, electrode, and spoken word. When sources of error were reduced as much as possible, correct computer classification rates ranged from 52 to 72%,
• EMG values performed better than EMG/EEG combined which performed better than EEG alone
• When templates of one subject were used to classify words based on individual responses of another subject, the percentage of correct classifications for EEG responses was no greater than chance expectation. The percentage of correct classifications for EMG was greater than chance, but not nearly so good as within subjects. Thus, each subject’s biological patterns associated with speech appear to be unique
• Performance would be better if known sources of error were removed such as time and amplitude variations and muscle and eye movement artifacts.
• Stored patterns should be “refreshed” periodically (i.e., new templates should be formed and updated) to take ac­count of “drifting” cortical organization
• The system performed best for subjects who have strong hemispheric lateralization for language.

In the end Pinneo concluded that

…it is feasible to use the human EEG coincident with overt and covert speech as inputs to a com­puter for such communication. However, we also conclude that, without additional research, the EEG is not adequate for the design of a prac­tical operating system; indeed, other methods than those employed here may prove superior. Nevertheless, enough information has been obtained during this proj­ect to specify the optimum parameters to use for an EEG-operating system and to suggest future research toward that end. Our results show con­clusively that consistent, repeatable patterns exist in the EEG during overt speech (for example, see Figure 19) and covert speech (Figures 13 and 14) and that a computer can recognize these patterns a statistically significant percentage of the time.

Although the work was commissioned by DARPA the work wasn’t exactly top-secret in fact Time published an article on Pinneo’s work in 1974:

Pinneo does not worry that mind-reading computers might be abused by Big Brotherly governments or overly zealous police trying to ferret out the innermost thoughts of citizens. Rather than a menace, he says, they could be a highly civilizing influence. In the future, Pinneo speculates, technology may well be sufficiently advanced to feed information from the computer directly back into the brain. People with problems, for example, might don mind-reading helmets (“thinking caps”) that let the computer help them untangle everything from complex tax returns to matrimonial messes.

This was definitely groundbreaking work and there is no doubt that it grabbed the attention of people within the defense and intelligence communities. In the non-classified world this work has been taken up Chuck Jorgensen of NASA who I’ll come back to in a future post.

 

References

1. Feasibility Study for Design of a Biocybernetic Communication System, L.R. Pinneo, D.J. Hall, 1975
2. Mind Reading Computer, Time Magazine 1974
3. Subvocal Recognition, Wikipedia