Technology and Emotions | Roz Picard | TEDxSF

Translator: Mark Freehoff
Reviewer: Denise RQ Thank you.
It’s a pleasure to be here. When something really matters it elicits emotion and how do we help people
communicate emotion, to make life better,
whether at home, at work, for your customers,
for your kids at school? Our challenge at The MIT Media Lab
and at Affectiva, has been to come up with better ways to measure and help people
communicate emotion. I’ll start with an example
that we did years ago which was a skin conducting sensor that we gave to an audience like you,
worn on the palm of the hand and it glowed brightly every time the audience got excited.
Every time they were engaged. We learned a lot that day. We learned for example
that every time a new speaker came out on stage,
the audience glowed. Every time there was a live demonstration, whether it worked or not,
the audience glowed. Every time there was live Q&A or laughter, the audience glowed
but unfortunately, every single time there was a PowerPoint presentation there was a decaying exponential
in brightness. (Laughter) Let’s, let’s look more closely at an example of the signal changing. Here, for a nine year old boy,
watching a movie trailer, the signal will go up about a second
after each of the parts of the trailer that cause him to engage. Starting with the music. (Music) The three biggest peaks. (Laughter) See this is where it
kind of falls of here though. (Music) He peaks for the name of the movie
and the date of it, so you can see why people
in Market Research and Advertising are extremely interested
in this information or I as an educator,
I’m extremely interested too. ‘Cause I want to know what content is connecting with my students
and with my loved ones. Here’s an example of the signal
changing with a Patriots fan, back when the Patriots
were in the Superbowl. And this one illustrates an important point
about skin conductance and that is even as we measure
the auto-immunic arousal of a person and their excitement. This goes up with things
that are good or bad. So we see it go up here
when the guy has a — when his favorite team
scores a touchdown, it also goes up
during an endzone overthrow. So it goes up with good and bad. Interestingly, it also goes up
during a Dorito’s Mouse T.V. commercial, when the mouse comes out
and catches the man. (Laughter) It’s interesting now that we can get
this information in real life, to see how relatively important
different events are in life and where the significance of something
that you share with somebody, compares with this response to some other things
that are going on with their life. What drives this signal? This skin conductance is driven
by the autonomic nervous system, sympathetic nervous system response. Your autonomic nervous system
has two main branches, the sympathetic and the parasympathetic. The sympathetic can be thought of kind of like
stepping on the gas, in the car, it revs your heart up, it causes
the skin conductance to increase. The parasympathetic causes
the heart to decrease. The sympathetic is thought of
as fight or flight, the parasympathetic is rest & digest. Almost all the organs are innervated by both
the sympathetic and the parasympathetic, except the skin,
which gets the pure sympathetic, which makes it ideal for us measuring
this fight or flight response. Now, I became interested in measuring this response
when I started working a lot with people in the autism spectrum,
who are non-speaking. The most heart-breaking stories,
that I hear from these — from the families
and the individuals themselves, when they later acquire
the ability to speak, is that, the changes that were worst,
were when they got kicked out of school or were shipped off
to an institution, removed from learning opportunities
and social opportunities. What caused that to happen? Usually it was a meltdown
or a series of meltdowns and the meltdowns, like adult tantrums,
appear to come from nowhere. One moment the person seemed
calm and relaxed, the next moment they might have become injurious to themselves or to others. So, they seemed very unpredictable
and uncontrollable. Now, as we undertook the privilege
of getting to know some individuals who could later communicate,
they told us that these meltdowns never came from nowhere
that, in fact, they were always preceeded by increasing stress and frustration. They couldn’t understand why people
couldn’t see what was bugging them. So this made me think,
“Wow, what if we could enable them even though they can’t communicate, to signal out their increasing stress
and frustration?” But to do that required a device
that is shown on the left here, a classic, galvanic skin response, skin conductance
sympathetic nervous system measure, that involved a lot of wires and boxes, and that was uncomfortable
to wear in daily life. So my team at MIT went through
lots of iterations until we came up with a device
that could be worn robustly on the wrist and gather data in real life. This has just been commercialized by Affectiva and I’m pleased to say
is finally available as a product for people to gather
and communicate this information in the real world,
outside the lab. Here’s the first example of seven days of data
that I saw from a student at MIT wearing the skin conductance sensor
on his wrist. And the first thing that jumped out here
was there’s a lot of studying and it’s very activating. Studying and lab work, in fact, sort of
to the discouragement of we, professors, is much more arousing
that any of the classroom activity. (Laughter) We also saw these huge peaks that were happening
during sleep which puzzled us in the first place. We just expected sleep to be flat. Generally, as we did
further polysonography, and teamed up with some expert
sleep researchers on this, we found that these peaks tend to occur during slow wave sleep or deep sleep,
and we’re now looking at the relation of those to sleep quality
and memory and learning. Here’s an example,
of a girl on the autism spectrum, where she’s able to communicate out
her arousal level, using this sensor. Here she’s wearing one on each ankle
so you’ll see two signals streaming across the bottom. She’s wearing it while she’s undergoing
an ordinary occupational therapy session. This is 45 minutes
of an ordinary session here, and you’ll see, you know,
it peaked here and she had a little bit of a meltdown here,
got in a ball-pit, calmed down, peaked here,
calmed down. What you see up here, in blue, is right now,
this window here the right edge is what you’re seeing in the video
and we see that it peeked as she climbed on this swing,
but as she starts swinging, as that rhythmic activity kicks in,
it has this nice, decaying exponential. We see for many people on the autisim spectrum,
when they rock, or do these repetitive movements they can be very calming for them. They’re not just doing that
to be difficult or something. It didn’t have a very important, regulatory effect. In fact, I’ve noticed an increasing number
of rocking chairs, cropping up in airports since 9/11 and they’re almost always taken,
at least at Boston Logan. I think that’s a healthy sign. Once you get the ability to make
a new measure in the real world and have people wear it,
unobtrusively in daily life, you learn a lot of things that are surprising
and this is a really special surprise we ran into. One of our kids on the autism spectrum
was wearing this over Christmas. I was looking at the data and I saw
this enormous peak on one side not the other side, I thought: “Oh dear, something
is going wrong with the sensor!” I looked more closely and it looked like
it was fine before the peak and after. I asked his brother, “Any idea
what could have happened here?” and he said, “I was with him
and he had a seizure right after.” Wow! I got on the phone with Joe Madson,
Boston Children’s Hospital, “Is there any way somebody could have
a huge auto-immunic surge on one side of the body
with a big seizure?” He said: “Yes” Fast forward about 8,000 hours of data
and extensive analysis, led by my student, Ming-Zher Poh
for his doctoral work and we have now shown that measuring signals
from the wrist, reflects signals measured from the brain
and the seizures are labeled by the epileptologists reading the EEG only,
they coincide exactly with these peaks, measured just from this electrical signal,
from the surface of the skin, on the wrist. So this was really mind blowing to me! That a change in our brain could show up as these events, picked up through a simple wrist band. We’re hoping this now can lead to better detection
and treatment in real time of people who suffer from seizures,
of the seizure events themselves. Now another thing that kind of surprised me
in doing this, was — the events show up,
sometimes, on just one side. Usually the two sides
are showing the same information. We where just measuring two sides
for redundancy, to cancel noise, but we found that — we learned something
that was already in the literature the right hemisphere of the brain controls
the electrodermal activity of the right palmar surface, and the left,
the left. I sort of rediscovered this when a tragic event
happened in my family. I lost a loved one and that day I was at work,
having trouble speaking. Those who know me,
know I never have trouble speaking. So, one of my colleagues,
Ron Elco, said: “Put a sensor on both sides!” I’m strongly right-handed and sure enough,
the left side was incredibly suppressed. That night I was, sitting in the kitchen,
in this total stupor, just kind a stirring a pot of something and one of my sons came up
and started rubbing my back which, I felt really blessed,
this is a very special moment, I almost felt hope restored and then, his little brother
came up and did the same thing to him. I had one of those rare moments of parental bliss,
where you kind of tingle down to your toes especially as a mom of sons. And —
(Laughter) at that moment, my little one looks up at the display
in the kitchen, ’cause of course I’m streaming my data
wirelessly to this monitor, (Laughter) and he says:
“Mommy, you’re lines crossed!” My right signal that was way above my left,
had come together during that moment of parental bliss. Now I thought: “Wow! I’ve heard about
positive and negative stuff for the brain you now might show up but I never thought
this could show up on the wrist. Now, that’s just a few data points,
but I monitored myself overtime, with sleep and sure enough,
the signal, as I recovered, went back to the normal
sort of switching, left and right. Now this is just a few data points but it shows
that there’s this whole new potential to explore what’s going on in the brain, even by measuring
some of these peripheral signals downstream. There are many positive moments as well,
that people have shared with us, using our emotion technology. Here’s one from a bride in India, who, naturaly,
was sort of stressed out before her wedding you see lots of peaks. Her friend loved seeing this part
where she was chilling with her friends and they helped calm her down before
and then the four major peaks here, during the wedding ceremony. The first, where the vows
were exchanged, the second, third and fourth
in the Hindu ceremony where the bride and groom
exchange symbolic gestures that they are now
husband and wife. Here’s an example where a mother
and a daughter wanted to share the daughter’s experience at school. The daughter had her first, big concert,
in front of school that day. She was playing a harp and singing
in front of the entire school and she expected it would be kind of
peeked there, and it was. Also it was very peeked during physical education,
to be expected anything that gives physical exertion,
makes you sweat, can be very arousing. Her mom was very disappointed
to see that math lessons, (Laughter) was the low point of the day. But there was this other, huge peak there
between Reading and PE, and — What was that? Turned out the daughter had a talk
with her mom, later that day. In fact as she recalled this experience, we see
the emotions coming back up again. This was this horrible, bullying event that she
was being dragged into by her friends. Wouldn’t it be amazing if the people
we care most about were given the opportunity
to share, in an externalized way some of what was going in their day
that was so difficult. Whether they can speak or not, this can be
a powerful way to take your emotions outside of you and begin to reflect on them
and share them with others. I have one last example with
the skin conductance sensor, a personal one. I discovered one day
while taking my son ona birthday outing to Six Flags,
that I indeed had big peaks with the Roller Coasters,
which I love, and a really awful one with a ride that
I won’t tell you about, that was very unpleasant. (Laughter) But the surprise to me, when I looked at the data
at the end of the day, was not that I had this huge peak when I’m
on the fastest, highest, most intense roller coaster in all of New England, but it was actually
earlier in the day when we were just trying
to get out the door. (Laughter) Many people on the autism spectrum
who are conversant, tell us that they have difficulty
reading facial expressions, they are trying to pay attention to what you’re saying
and at the same time, they can’t tell if you’re looking pleased
or displeased, interested or bored. In particular, many of them have asked us
if we could please help them to have some technology
that tells them if the person they’re speaking with is still interested
or if they’re boring them. So this next example is a short video clip,
introducing my collaborator on this work, Dr. Rana el Kaliouby, who’ s also
my co-founder at Afaktiva and the chief architect
of our face reading software. “– Which would be recording your facial expressions
and analysing them in real time. So we use 24 feature points, on the face,
like the the eyes, mouth, eyebrows, and we use those to recognize facial expressions
like, a smile, a lip pucker, eyebrow raise and a mouth open
and so on. And we look at these over time
and we sort of use them along with head gestures to make a guess
at your hidden understatement. (Video) Each person is wearing a camera,
pointing at their own face often. (Video) (Laughter) All right. Using this technology,
there are many things we can recognize and many things we can learn, for example
I learned that actually, I was quite interested in what he had to say, but I started
to feel dis-interested because he never paused, he never gave me a chance
to go into thinking mode, process the information and return. Or when I did, he had moved on
and I was lost. So as we do this fine grain measurement of emotion,
we learn a lot of things about how to improve our interactions with one another
or how to improve our interactions on-line or elsewhere in the world. Here is just an example reminding us that, you know,
while emotion is about what matters most, it’s also communicating this complicated language,
that is non-verbal. In order to get a computer
to learn how to read this, we have to solve an extraordinary,
complex problem that makes chess and Go
look trivial in comparison. With over ten thousand facial expressions
that can be communicated and changed, every second, multiplied by the faces
participating in a conversation, the social, emotional games people play
is probably the hardest computer science problem to solve,
in the next century as we go forth. The complexity is just astronomical. What we are starting to work on at MIT
and in partnership with Affectiva, in this next project, is a way to go online, to get right now
thousands and hopefully, before long, millions of examples of natural facial expressions,
while they happen online. Here’s an example where we partnered
with an interactive task on Forbes. You can go to
and try this out. You click on one of the T.V. commercials.
These are designed to elicit delight and we measure your smiles
and we show the results here for one of the commercials,
for three groups of people. Those who say they have seen
the video many times, those who have seen it once or twice
and those who say this is their first time. This should be playing. But for some reason it’s not playing.
Clicking! And what will happen
when they get there and play this video is you’ll see the video synchronized
with the peaks, on the graph. You can go online and see this,
if they can’t get it working here. If you go to
and search for interactive smile you can participate, give our researchers
your facial expression data. You can also see the results
and see interesting things, like, you know, “Do people smile where you smile,
if you smile at the dog?” One thing we learned is that people
who have seen it before smile much earlier in anticipation
and they also smiled a lot more, for these popular videos that they like. I seem to have lost my slides completely here.
I will wrap up and just tell you whether the emotions
are online or in real life, whether they’re to help somebody
who is speaking or non-speaking or just to capture some
of those most positive emotions, moments in life, new technology, is enabling people to communicate emotion,
in ways never before possible. I’d love to hear from you, how this technology
could help improve lives for you, your business, your customers
and your loved ones. Thank you. (Applause)

, , , , , , , , , , ,

Post navigation

30 thoughts on “Technology and Emotions | Roz Picard | TEDxSF

  1. I have a theory that may be very interesting to you. In short it hypothisises that tone of voice is music because it is universal across language, like facial expression, instrumental music being like puppetry where inanimate objects are manipulated to seem to express emotion. Working backwards from music I have formulated a likley "periodic table" of emotional elements that would then be the basis of all emotional compounds. See my TEDx video "The Geometry of Emotion"

  2. Cultural convention sets limits on EVERY human activity but cultural convention can NOT change blue to red bitter to sweet or high to low. The basics of sensation are INHERENT in physiology and not amendable by culture, Further studies on tone of voice and facial expression do not show them to be cultural specific, rather these have been shown to be recognizable across cultural barriers, You can verify this yourself by watching a foreign film, emotion is easy to read even if language is unknown

  3. By the way not only you can tell what a character feels in a foreign film when you do not know the language and customs with your EYES CLOSED so that you can ONLY hear tone of voice or else with your EARS COVERED so you can ONLY see facial expression.What IS prescribed by culture is what emotion is OK or is not OK to feel in a given situation. How a given emotion is manifested in tone of voice or facial expression seems to be like breathing, it is instinctual and universal.

  4. People pretend to have feelings they don't have, people pretend not to have feelings they do have, its a mistake to take all that at face value however and declare foreigners "inherently different" because intolerance can ride in on this idea, and would be difficult without it. Anyway it all boils down me asking "check out the theory I've developed in 25 years of study" and you saying "I don't need to check it out because its wrong" a pointless exercise in futility, I'll not write again.

  5. I guess that is true, I was more thinking about body language. But what is true is the intensity of facial and tonal expressions varies with culture. For instance, as a westerner, in general, I have a harder time reading the facial expressions of Asian people. And at the risk of sounding offensive, some Asian languages sound angry or "stressful" a lot of the time just because of their rhythm and sounds.

  6. no look up unviresal expressions happyneiss sadness etc are all the same for all cultures paul ekman dicoverd this

  7. plus, I think there was a study, measuring American's and Asian's (Japanese) different way to express emotions. Of course Americans were more expressive than Asians, BUT the interesting fact was that this gap between cultures was limited to social scenarios, while there was no difference in showing emotions when the American or the Japanese were by themselves (like, at home, watching tv, …).

  8. The Affective sensors are breakthrough innovation. With simple measurements of temperature and perspiration, the sensors can correctly predict positive and negative responses to stimuli. Combining it with facial emotion detection is only an approximation. Where temperature and perspiration are physiological, sentiment is subjective. With machine intelligence those approximations will significantly improve.

  9. What about the noise?? people here are moving the GSR or EDA could be higher just because they are moving…

  10. Very motivating speech but i have to point out a few things. Firstly, you cannot do electrodermal activity without having a solid theoretical framework to work with. What Picard does here is to re-invent the wheel. Where are the extensive works mentioned that explain how SCL works and how it is connected to arousal and stress and emotion states? It is not like there is nothing here. She could have mentioned Fowles, Boucsein. She could have mentioned Arousal Theorists. The problem is also that without a theoretical framework you just do not know why the peak is happening. And so it does not explain anything, really. I realize that you cannot cite dozens of names in such a presentation, but a FEW names would have been helpful and shown that there are others who worked on skin conductance and Arousal.

  11. Prof Picard — what is the likelihood of a system that can make qualitative discernment — High (fear vs. extreme joy), Low (boredom vs. peacefulness)… Thanks for this great info!

  12. When I'm sad or stressed, when anything's off basically, I can feel it in my wrists. My right one, mostly. Kind of makes sense now. It's like joint pain, but deeper. Idk if that's something different but that's what I thought when I saw the left/right wrist graph

  13. This technology would be amazing for Aspies and people with mental blocks. I have a difficult time understanding facial expressions or interpreting ques and I have major social anxiety where at times, I cannot stand still and almost go into a frenzy of anxiety.

Leave a Reply

Your email address will not be published. Required fields are marked *