Lipreading.

Another interesting post by Dan Nosowitz at Atlas Obscura; it’s called What Is the Hardest Language in the World to Lipread? but that’s something of a bait-and-switch — the URL’s “lipreading-around-the-world” is much more accurate. Nosowitz starts off asking “What language, of the 6,000-plus distinct tongues in the world, is the hardest to lipread?” and continues:

This last question, though seemingly simple, resists every attempt to answer it. Every theory runs into brick walls of evidence, the research is limited, and even the basic understanding of what lipreading is, how effective it is, and how it works is laden with conflicting points of view. This question, frankly, is a nightmare.

He then goes into a long and fascinating discussion of how lipreading works; he talks about “profound connection between the auditory and visual senses when understanding speech” and explains the McGurk Effect, then getting around to lipreading itself:

González does not know American Sign Language. She relies entirely on lipreading and the little bits of sound she can detect with her hearing aid. Our conversation was very smooth […] The commonly cited low success rate for lipreading isn’t based on people like González. Much of the research uses subjects with normal hearing; Masapollo, for example, doesn’t work with deaf subjects. His work is less about lipreading than it is about the perception of speech. In fact, it’s weirdly difficult to find data about the effectiveness of lipreading as practiced by people who actually do it as part of their daily lives. People with normal hearing test at around 10 to 12 percent accuracy, and there’s some suggestion that hearing impaired people are more like 45 percent accurate. But most of these studies are a little weird, because they tend to test accuracy by seeing whether people can identify individual phonemes—the sounds that make up words, like “gah” or “th”—or words. But, according to González, that’s now how lipreading works.

“We don’t lipread sounds,” she says. “Some people think we’re looking at phonemes and stringing them into words. Doesn’t work anything like that. We’re seeing words and putting them together in sentences.” Context is everything for a lipreader, because, well, context is everything for anyone trying to understand speech, regardless of which sense is being used. Whether you can trick a lipreader into confusing the words “ball” and “bull” is not reflective of real-world accuracy, because it’s unlikely that those words would be used in a way that makes it unclear which one is intended: “The dog wants you to throw his bull.”

Not many studies acknowledge any of this, though one from 2000, in the journal Perception & Psychophysics, did. In it, the researchers examined lipreading abilities among those with impaired hearing (IH, in the study) and normal hearing. Crucially, it was assessing the ability to understand entire sentences rather than just phonemes. “Visual speech perception in highly skilled IH lipreaders is similar in accuracy to auditory speech perception under somewhat difficult to somewhat favorable listening conditions,” the researchers concluded. This tracks with what González told me: Lipreading, once you get really good at it, is more or less equivalent to understanding a speaker in a busy restaurant.

Eventually he gets around to the initial question:

There’s a body of literature on lipreading various non-English languages, but they don’t make things any clearer. The problem with trying to find the hardest language to lipread is that lipreading is exceedingly based on the individual, and most of the best lipreaders in each language—the ones who depend on it—don’t show up in studies. So figuring out how things differ from language to language requires an awful lot of guesswork. […] My first thought was to figure out which phonemes are especially hard to distinguish, under the assumption that a language that has more of those must be harder to lipread, right? […] Another element of some languages that could present lipreading difficulty comes from tones. […] Still other languages heavily rely on another class of sounds made without much from the lips—namely, by using the larynx or creating a sound back in the throat. […] I’ll toss in Chechen, because it has such a startling number of consonants and vowels, around 50 of each, including a whole mess of sounds that are made back in the mouth and throat. On the opposite end of the spectrum is Pirahã, a language spoken by a few hundred people in the Brazilian Amazon. Pirahã has about as few phonemes as a language can possibly have, which I suspect would make a lot of words look the same. It’s also a tonal language and may be substantially informed by rhythm; nobody’s quite sure […]

There’s no good answer to the question, but then again, there are very few good answers to any questions about lipreading.

More details at the link, obviously; it’s interesting stuff to think about, even if there aren’t many answers.

Comments

  1. ‘individual phonemes—the sounds that make up words, like “gah”‘

    I presume “gah” is meant to represent /g/ ; Nosowitz probably knows what a phoneme is, but he’s not helping his readers with that adhoc respelling

  2. Old school British RP must be difficult, since the upper lip doesn’t move at all and the lower lip as little as possible.

  3. Jonathan D says:

    about as few phonemes as a language can possibly have, which I suspect would make a lot of words look the same.

    My naive intuition would suggest the opposite…

  4. David Marjanović says:

    and the lower lip as little as possible

    …but Britons *actually* *move* their *jaws* when they speak.

  5. Indeed, but jaw movements are crude and hard to interpret

  6. David Eddyshaw says:

    I wonder if languages that don’t go a bundle on lip rounding are particularly difficult (like Japanese; a fortiori, Mohawk, though I imagine that might be a bit harder to investigate.)

    Pirahã has about as few phonemes as a language can possibly have, which I suspect would make a lot of words look the same.

    Unless several Pirahã phonemes happen not to be very different for lipreading purposes, their fewness is surely quite irrelevant: if words are sufficiently distinct to be distinguished adequately by hearing, they’ll be just as distinguishable by lip-reading. Indeed, the fewer the phonemes, the longer the words on average, which is going to be helpful either way.

    I would agree that languages in which tone carries a heavy load (which may include Pirahã) are likely to be hard.

  7. The best teacher I ever had, in any subject at any level, was totally deaf. This was the late H. Latham Breunig,

    https://liblists.wrlc.org/biographies/54246

    who taught an in-house statistics course at Eli Lilly & Co. There, the only concession he made to his handicap — the only concession — was to ask us to stand when we asked questions and look him directly in the face so he could read our lips.

    Of course, in a class dealing with a technical subject the vocabulary is restricted and the general context is understood in advance. Even so, on the basis of my experience in the Breunig classroom I think I’d agree that the claim of 45 percent comprehension among deaf lipreaders

    (“People with normal hearing test at around 10 to 12 percent accuracy, and there’s some suggestion that hearing impaired people are more like 45 percent accurate. But most of these studies are a little weird, because they tend to test accuracy by seeing whether people can identify individual phonemes—the sounds that make up words, like “gah” or “th”—or words. But, according to González, that’s now how lipreading works”)

    is a little weird. Dr. Breunig seemed to understand everything that was said, with no need at all for paraphrase or repetition.

  8. I was puzzled by this from the article: “In countries such as India and China, with high illiteracy rates among the deaf and reduced resources for teaching, oralism is still popular”

    Why would higher literacy in English cause more use of ASL over lipreading? (The reverse appears to be the case, though, that children acquire English literacy more easily if they and their parents have ASL i.e. a shared spoken language to read with.)

    Would “reduced resources” simply mean “not able to educate in sign, so students learn to lipread or nothing”? That mechanism makes sad sense. (Why sad? Well for one thing, it’s largely one-way communication.)

  9. “As far back as the 50s, there were classic studies that showed that people are better at perceiving speech in the presence of background noise if you can see the face of the talker,”

    Duh. You don’t need classic studies to tell you this. Speaking to person in a foreign language is harder over a telephone than face to face.

  10. Actually, you do need classic studies to tell you the apparently obvious, because sometimes the apparently obvious turns out to be wrong.

  11. Eli Nelson says:

    Telephones have the additional factor of bad audio quality, which is probably easier to deal with for one’s native language.

  12. I wonder what kind of a difference “hearing impaired people” (in the studies) vs totally deaf people makes a difference as far as the accuracy of lipreading. Or said differently, do different people within “hearing impaired people” have different degrees of ability at lipreading?

    And a lipreading anecdote. I’m firmly in the hearing camp. But I do recall once seeing a rock band playing live, where I knew some of the musicians, and one of them said something to me from on stage, and I totally had no problem knowing what he said when he said it. Only a little later did I realize I had not heard him at all and my sensation of having heard him, and my knowing what he said, was based on lipreading. Alas, my ability to understand when the environment makes hearing difficult or impossible usually isn’t that good. But that affirms for me that fully hearing folks, who can hear when we can’t see, can and do sometimes use lipreading.

  13. David Marjanović says:

    Hollywood routinely, and to me very noticeably, assumes that everyone can lipread, and that everyone shares that assumption. Indeed the English language has a word for making that assumption: to mouth words – to articulate with exaggerated movements for the purpose of communicating a word or two without sound in bad acoustics or while talking to someone else on the telephone.

    I’ve only seen one person try this in German, and that was the classmate who watched so much TV that Hollywood was basically her native culture. I don’t think it ever worked.

  14. John Cowan says:

    There is also ventriloquism, which depends on anti-lipreading. Basically all labial consonants are replaced by velars or in some cases coronals. Here’s an example from James Blish’s fixup novel Cities in Flight: “Yatch Heldon, Karst. See yhich syitch he kulls, an’ nenorize its location. Got it? Good.” Here all the labials become palatals except /m/, because “ngengorize” is phonotactically impossible.

  15. David Marjanović says:

    So ventriloquism sounds like talking while brushing your teeth? That’s a disappointment.

    (I’ve never encountered a ventriloquist.)

  16. David Eddyshaw says:

    Angrose ghishock!

  17. Typically ventriloquists assume funny accents anyhow; the ventriloquist is the straight man, the dummy is the comic. YouTube clip of Edgar Bergen and Charlie McCarthy.

    But I have to say that mouthing words, unless you are an actor, is more a conceit than a reality. The few times I have tried mouthing words because I really needed to communicate only to my partner and not to anyone else failed miserably.

    Per contra, it is much harder for me to understand someone who has a different accent or who mumbles when I can’t see mouth movements. The style I often see on BritTV of shooting the listener in close-up so we see their reactions makes understanding the speaker much harder. But as Gale’s lost about 13% of her hearing, we keep subtitles on anyway.

  18. David Marjanović says:

    I put myself in front of a mirror today. Unrounded vowels = mouth at all visibly open, rounded vowels = lips relaxed, not rounded, mouth looks closed unless you look very closely; the rounding is entirely outsourced to the tongue unless I’m speaking French.

  19. Lars Mathiesen says:

    Well, that’s ‘compressed rounding’ — it’s not an active gesture as much as an opposition to ‘spreading’ where the lips are stretched out. Did you test with front or back vowels?

    From the couple of languages I found the information for, front rounded vowels often use the compressed version and back rounded vowels are protruded which is much more visible. The terms ’rounded’ and ‘unrounded’ fit better with back vowels, I think.

    (Impressionistically (listening to myself), compressed rounding sounds ‘between’ spread and protruded, but for front vowels compressed and protruded are very similar (unless you’re Swedish, I guess), while for back vowels the extra effort for protrusion really makes a difference).

  20. David Marjanović says:

    Did you test with front or back vowels?

    Both. At first I wanted to link to the WP article on vowel rounding, but the photos of both “protruded” and “compressed” rounding are way beyond anything I normally do, so it would have been misleading.

    Did you test with front or back vowels?

    Both. LIke, there’s a millimeter of lower-lip protrusion going on when I suddenly switch from [u] ot [y] – if you don’t watch it, you miss it, and if I come to [y] from somewhere else, it can be absent altogether.

  21. Lars Mathiesen says:

    So your rounded front is more protruded than your rounded back, or did you mean from [i] to [y]?

    In any case, that’s why Anglophone photographers want you to say cheese — the spread lips and closed jaw make for at least the start of a smile. Danish smil is almost self-describing, lips are tensely spread throughout the last three segments.

  22. Hollywood routinely, and to me very noticeably, assumes that everyone can lipread

    From an article by William Safire, in 1988. Advertising slogans & Hollywood scripts (“Where’s the beef?” and “Go ahead, make my day!”) were grabbed by speechwriters in American presidential elections during the 1980s. Then they stopped, I think.

    Read my lips is rooted in rock music. In 1978, Tim Curry gave that name to an album of songs written by others. He got the phrase from an Italian-American recording engineer: “I would say to him, ‘We got it that time,’ and he would say, ‘Read my lips – we didn’t.'” And what is Mr. Curry’s definition of >read my lips? “Listen and listen very hard, because I want you to hear what I’ve got to say.”

    I suppose a recording engineer is often on the other side of the glass from the performer.

  23. I’ll toss in Chechen

    Аудиокурс чеченского языка

    A course in Chechen (explanations in Russian)

  24. David Marjanović says:

    So that’s what it looks like when I’m not legally sane in the morning…

    I just tried again. No lip movement whatsoever from [u] to [y], to my own surprise. From [i] to [y] all that happens is that the lower lip relaxes, i.e. moves back up and leaves just a narrow slit ( ~ 1 mm) of greatly variable length, meaning it’s not necessarily round.

    I’d need to be filmed while talking, and that in different languages.

  25. The most famous appearance of, “Read my lips,” prior to George Bush’s pandering to the anti-tax crowd in New Hampshire, was from Beverly Hills Cop. Eddie Murphy says, “Read my lips: five thousand dollars,” to one of the crooks who is trying to stiff him in the opening sting.

Speak Your Mind

*