Connecting Emotions to Extreme Sports & Next-Gen Media
The Art & Science of Measuring Emotions in Extreme Sports, and how it applies to the Future of Media Experiences and Society.
First published on Medium, this is an in-depth look at the processes and outcomes of our project. The original project blog post is here.
In early 2016 I was handed a project brief that I instantly knew would consume my life for the foreseeable future.
Red Bull Media House had asked us to figure out how our technology could be used to create a new level of emotional connection between media audiences and the extreme-sports athletes they follow. We framed the entire project with the question, ‘how does it feel to be a Red Bull Hero?’
When the brief landed on my desk, my brain instantly ran in two opposing directions:
‘Holy shit, this is going to be the coolest thing I’ve ever worked on!’
‘Holy shit, there are a lot of ways to screw this up’.
If successful, we would have pioneered a novel form of immersive entertainment, broken new scientific ground, solved a ton of technical problems, and prototyped a set of innovative tools for empathic human-machine interaction. And we would have done so in partnership with one of the slickest brands on the planet.
But there were sooooo many things that could go wrong. As a taster, how about:
- Wiring the athletes up to physiological sensors without distracting, injuring or killing them.
- Extracting a clear data signal from unreliable wireless tech while it is being hammered by vibration, moisture, cold temperatures, excessive transmission distances… you name it.
- Handling a huge pile of data, HD video and audio — for recording, storage, processing, communication and analysis.
- Identifying emotionally triggered biometric changes (eg. a rise in heart rate) while the performers are already in a state of heightened exertion.
The outcome was The Hero Feeling, a prototype world-first ‘bio-emotional’ VR experience, and Sensum SYNC (now replaced by newer solutions) – a suite of new beta tools for ‘bio-emotional’ productions. They’re all free so please dive in and play with them.
Now that we have finally been able to release some of the fruits of this endeavour, I want to share some lessons learned from coordinating the project, and take a brief look at the future of empathic media.
The Three ‘S’s of Empathic Technology
We have an in-house framework we apply to complex projects like this. Well, maybe nothing so structured as a framework — more like a motto:
Science > Software > Story
And it always starts with the science.
Measuring emotions is still an exploratory field, especially in the field, beyond the luxury of lab conditions, out there where people live and breath. Collaborating with our friends across the road from us at the School of Psychology, Queen’s University Belfast, we set out to build a model of the emotional journey an ordinary person takes in an extreme activity. Then we would have a base model, plus the insights gathered from our field trials, that could be developed for elite athletes performing extreme sports.
First stop was at the sports science lab at Ulster University, sampling athletes’ biometric responses to some simple exercises, to establish our base and gain some real data on how people’s bodies respond under exertion.
Next we wanted to build a generalised model of extreme emotions, that could be applied across various activities. So we threw 40 members of the public down a zipline and round a 4x4 racetrack at an outdoor pursuits centre. I know, fun.
It was a gamble. There doesn’t seem to be much relevant literature to lean on, and we were out in the cold Northern Irish wild with a load of unpredictable kit and no certainty of finding any significant data. What could possibly go wrong?
Well, pretty much nothing in the end, because we planned the crap out of it. Nevertheless, one thing that nearly killed the experiment was believing that students would get out of bed to participate in it. Consider: we were offering to pay, feed and transport them for a day-trip to an adventure centre. Not a bad offer. But two-thirds of those who registered didn’t show on the day.
This is why you have to budget for redundancy and screw-ups. As the Navy Seals say:
‘Two is one and one is none’.
Having over-planned and over-subscribed, we got a full quorum of 40 participants. And the experiment worked. Half of the group did a short warm-up exercise before each activity, raising their heart rate to give us an ‘exertion’ state, for comparison against the non-warmed-up half. In the end, we could see emotional data, primarily in heart rate and skin conductance, peaking over the fuzzy noise of both exertion and environmental factors.
So far so good.
It was around that time that our main advisor on emotions science, Dr Gary McKeown from Queen’s University, was musing on how good the data would be if we could put an amateur and an expert through an identical and simultaneous experience. We could compare their biometric and emotional responses to explore the experts’ ability to control their feelings under pressure. He suggested tandem skydiving as an example. We’re so glad he did.
Not only did we get to leap out of a plane 4,000m up in the Irish summer air, but the manager of the airfield had an additional suggestion. ‘If I’ve understood your experiment correctly, it might also work in a stunt plane’, he said. So we also got to ride with a European aerobatics champion, as he put us through aerial manoeuvres at up to 8g.
Damn, Science, you never told us you could be that much fun.
Incorporating the scientific method into your client projects can slow down and complicate your service delivery (read: increase budget). But there’s an inherent value in approaching your work with a scientist’s skepticism and taking a first-principles approach to solving problems. You never know what you will get to discover or invent, and you build a foundation of knowledge that you can stand by in future.
Not only did we get the model we wanted for our extreme sports context, we discovered some intriguing evidence about human physiology and psychology as we went, some of which we’ve already been able to publish for others to build on.
Software: Inventing the Tools for the Job
The technological challenge was daunting. We needed to build a mobile app that could simultaneously control all the sensors around the athlete: biometric (eg. heart rate, skin conductance), media (eg. video, audio) and contextual (eg. location, speed). It would then need to record and synchronise the incoming data from all those sources and tag them with key events. Lining up metadata with media content is a long-standing problem in the production industry so we had to invent a new product to solve it.
OK, three new products: the mobile app, the data pipeline to connect with our cloud-based platform, and a media production plugin to make creative use of the processed data.
The simplest word for the overall software build was ‘sync’, and the name stuck. Hello, Sensum SYNC.
As with most of the emotion-measurement projects we’ve worked on, building the tools was only half the battle. In the field, if the software gives you trouble (which it always does) the hardware will find ten times more ways ways to ruin your day. Bluetooth connection is usually top of the list. Different Bluetooth devices have their own idiosyncratic ways of making and dropping connections, exacerbated by anything from the signals of other devices to weather conditions, to… ah hell, Gremlins? Who knows?
If you can record your data on your mobile device, to mitigate for the case of communications loss, do it. In fact, record it on multiple devices just in case. And contextually tag the shit out of it: start/stop points, key events, etc.
Furthermore, capture more data streams than you need. It’s hard to know which ones will give a good signal on the day, or which ones will provide unexpected insights. This is similar to how we approach emotion measurement in general: compounding multiple data streams to reduce uncertainty and increase reliability. It’s basically how us humans interpret the world too: paying attention to the mix of sensory input that is most available and appropriate to the situation.
Story: Telling an Emotional Tale
At our company, Sensum, we attempt to stay balanced while straddling the creative boundary between the art and science of empathic technology. We keep a foot buried in hard science while recognising that cold biometric data is worth little without a story to explain it. Narrative brings the data to life. And emotion lifts the narrative to a more human, immersive level.
Emotion gives the story wings.
Since launching the world’s first emotional-response horror film at South by Southwest (SXSW) back in 2011, we have exploited every opportunity to explore new forms of storytelling through emotion AI and empathic technology. For this project, the last of the three ‘S’s — the story — materialised primarily through the production of The Hero Feeling VR experience.
Empathic media content can broadly be divided into two types: one in which the emotion of the audience is measured to feed back into the media experience (like we did in 2011 with Unsound), and the other where it is the emotions of the actors or scenes that are connected to the audience in some way. For The Hero Feeling we wanted to trace the emotional journey of the ‘Hero’ — the athlete performing in front of the camera — and relay to that to the viewer. The viewer would get to ‘feel’ what it’s like for an elite athlete hammering down a mountain-bike trail.
To get our content, the project culminated in a three-day exercise we call a ‘data shoot’, combining various sensor tech with movie production. This time it took place on a bleak mountainside in Ireland, in December. Brrr.
We found four professional mountain-bike riders to be our heroes. The riders and their bikes were wired up with sensors measuring a range of biometric, contextual and media signals, including heart rate, breathing, skin conductance, skin temperature, location, speed, voice audio, ambient audio, contact audio from the bike frame, and PoV video.
We extended our scientific approach into the data shoot, controlling the conditions of each run down the two bike trails under expert supervision, and repeating each run to collect robust data. At the same time, we shot the event in HD (flat), while Red Bull sent over a team to shoot in 360º format.
For the resulting VR video, we overlaid visuals showing some of the data that was recorded from the mountain biker, using our custom-built Adobe plugin to manage the process. This includes a visualisation element that runs from Vulnerability to Control, which demonstrates the rider’s emotional control in this context. Control (also referred to as power or dominance) is one dimension of scoring used by psychologists to place emotions in a multidimensional space, with scores like arousal (how excited or relaxed you are) and valence (how positively or negatively you are responding) on the other dimensions.
If you were expecting labels like happy, sad or scared, I’m sorry to disappoint you. We often avoid that kind of ‘discrete’ labelling, as it can be inherently vague, subjective and misleading. It’s also not very relevant in the kind of ‘iceman’ scenario you get with trained performers keeping their cool while doing something they’ve done thousands of times before. Moreover, an activity like solo mountain biking is not a socially communicative context so we wouldn’t expect much expression of the kind of things most people think of as emotions, like smiling when you’re enjoying yourself. It’s worth pointing out that this is a very common issue, as we spend much of our lives engaging in activities that do not lend themselves to social communication, such as driving, consuming media or using a computer.
In the end, what the viewer gets from The Hero Feeling is a point-of-view experience, seeing the mountain through the eyes of an elite biker, with a synchronised feed of biometric, contextual and emotional data. Plus we introduce dynamic audio effects, the internal sounds of the bike, as well as other little bonuses.
Emotion is a fuzzy topic, clouded in misinterpretations and bullshit presented as science. We needed to walk through these extreme environments before we could run in them. Only then could we build the wings to fly.
Welcome to a World Where Media Can Feel
Examples of empathic media are emerging all around us.
Already people are experimenting with things like music playlists that respond to your current mood. We have done a lot of research that examines the precise points at which we become more or less engaged while consuming content, in order to edit it or change its delivery method for optimal impact.
In gaming, where narrative can be adjusted on-the-fly more readily than in media like video, your emotional response could control the experience you are presented with in real time. In our own trials of this concept the results have been immediately impressive.
As we increasingly allow machines to read our moods, we will see our interaction with media shifting. It’s likely we will soon be connected to the content that produces the strongest emotional responses, either in the performers, their audience, or ourselves. I suspect we will freely hand over our emotional data in return for more emotive media experiences, just as people share their heart rate with fitness apps now. But in so doing we are for the first time communicating some of our hidden, nonconscious emotional states with both software and each other. This is unprecedented in the evolution of our species.
Pushing our emotional buttons has always been the aim of media creators. We want to go on an emotional journey with our heroes, empathising with their experience without actually having to go through the pain of it. Now that we can measure the audience’s emotional response in a more immediate and granular fashion, we can tweak the content to push them further and faster. Ultimately, we may be able to induce desired emotional states on command.
Along with the development of empathic media, we advance our ability to manipulate our audience. Done right, this could be delightful. But it also opens doors for exploitation. Media producers share a responsibility to walk this road hand-in-hand with their audience, practicing transparency and consent.
With the right ethical norms in place, we can look forward to a near-future in which we get to join our heroes on their emotional journeys.
Special thanks to the wide range of people and organisations who contributed to this project, your creativity and energy is an inspiration. Apologies to anyone I've missed out: