Article 9 (Part 1 of 2) | Apr. 26, 2017
Understanding Athletes’ “Skill” and “State of Mind” with Virtual Reality Technology (Part 1)
Using highly-realistic VR for brain science measurements
One of the themes of the Sports Brain Science Project is research using virtual reality (VR) to study responses of the mind and body. What are the responses of head mounted display-wearing participants when they are shown highly realistic video of pitches? We spoke to Senior Researcher Dr.Toshitaka Kimura, who seeks to elucidate brain processes by using VR and explore the essence of winning in sports.
VR that fills the gap between the lab and the field
—The Sports Brain Science Project seeks to understand the essence of “skill” and “state of mind.” Dr. Kimura, tell us about experiments using virtual reality (VR) technology, your area of focus. Why use VR?
Kimura: Until now, the main approach to measuring in sports brain science is to measure in a laboratory environment. The biggest advantage to this approach is that measurement conditions can be rigorously controlled. But the sports activities and conditions that can be handled in a lab are far removed from real sport settings. So only limited aspects of sports science can be apprehended.
In recent years, as wireless sensor technology and wearable sensor technology grow in diversity and become more advanced, it has become possible to measure people’s behavioral information in the field as they engage in sports. At present we are using different types of wearable sensors like the conductive fabric hitoe® to obtain and analyze information from real sporting environments.
However, the closer you get to the real environment, the more complex the environmental conditions become. It becomes difficult to understand the factors that affect the phenomenon being observed. So we are constantly faced with the tradeoff of either getting reality-based results with real environment measurements or getting control of conditions with lab environment measurements. It is a quite a dilemma.
How can we fill the gap between lab measurements and real-world measurements? I thought we could utilize a third approach: measurements with VR. One of the advantages of VR is that it provides highly realistic visual experiences. What’s more, you can manipulate the visual environment in a variety of ways by combining VR with computer graphics (CG). In short, you can intentionally create visual conditions not possible in the real environment. So measuring with VR is brimming with possibilities.
VR technologies and services have already been introduced in places like amusement parks. They are used in attractions to provide roller coaster thrills and high-altitude sensations. In the field of sports, systems that can provide experiences in American football, kickboxing, tennis, and baseball batting are being developed. Researchers have begun using VR to evaluate sports performance.
What is VR for brain science experiments?
—At the “NTT Super Future Grand Expo” booth at the Nico Nico Super Conference held last April, visitors could experience VR that is the foundation for Sports Brain Science experiments. The experience was highly realistic!
Kimura: The VR system used at the Nico Nico Super Conference was developed by NTT Media Intelligence Laboratories. The HMD provides a highly immersive experience. Pitching scenes are projected to the HMD, and users can view pitches from the perspective of a poised batter or a crouching catcher. With the technology of this VR system as the foundation, we developed a batting system for measuring body movements. For the experiment, participants wear motion capture suits embedded with inertial sensors. We seek to study brain functions by measuring body movements and biological signals (muscle movements, heart rate, and respiration).
Figure 1. VR system for experiencing baseball batting
—Was the VR system developed specifically for this research?
Kimura: Yes. We developed a new system that had sufficient specs for measuring the human body and presenting reality. The original HMD system was 30 hertz (Hz). That is, it could only measure movements 30 times per second. That was not enough to observe behavioral response in detail. We developed a 90-Hz system, which has three times as much accuracy.
By the way, the virtual baseball field was constructed by combining real images and CG. This technology was developed by NTT Media Intelligence Laboratories. It allows an image to be reconstructed from an arbitrary perspective based on images taken by multiple cameras.
In this way, images as seen from the batter’s box can be reconstructed, even if the original images were taken from the spectator stands farther behind the batter’s box. Appropriate visual images could be presented independently to the right and left eyes of the user in sync with the position and direction of his or her head, providing participants with an extremely realistic batting experience.
—Why can’t you just use images taken from the perspective of someone standing in the batter’s box?
Kimura: Why do we have to purposefully reconstruct video? If we just record video from a camera attached to the goggles of a batter, only the conditions when the video was captured are real. For example, if the video is presented to a left-handed batter, he will feel a sense of disjointedness. In extreme cases, he will feel nausea like being carsick. So that the subject won’t feel discomfort no matter his position, reconstructing images by adjusting and processing the data is essential. The trajectory of the ball is also depicted by CG based on data of pitches obtained beforehand. It is projected in sync with the captured movements of the pitcher.
The movements of the batter and the swing of the bat are also reflected in real time to the avatar in the virtual environment. This is made possible by embedding wireless acceleration sensors in the user’s suit and bat. In short, when the user swings the bat and hits the ball, the ball flies through the air of the virtual ballpark in accordance to how it was hit by the bat. In this way, extremely realistic CG video could be produced in real time by seamlessly combining real images and CG. I think the most significant point of our system is its ability to realize an environment that meets the demands for measuring behavioral responses.
It is also possible to experience actually impossible conditions
—When I experienced the VR environment at the Nico Nico Super Conference, I unexpectedly fell backwards when a ball came flying straight toward my face.
Kimura: You were taken aback, weren’t you? (Laughter.) Even those of us used to the VR environment will fall backwards in spite of ourselves. This is because it is too realistic. If though the environment is not real but virtual, the reason we avoid the ball is because of our “unconsciously fast reaction”—that is, because of the workings of implicit brain function. Even if we think we can stop ourselves, we react automatically. Such a behavioral response is exactly what we seek to study.
What’s more, because the environment is virtual, we can present “beanballs”—pitches flying straight at the batter’s face—to the user. In the real word, it is not ethically permissible to throw beanballs at participants and observe their reactions! With a virtual environment, we can also show pitches with physically impossible movements, such as a ball that suddenly disappears or curves in an extreme manner. We have begun to study behavioral responses by presenting images with a variety of changing conditions.
Responses between baseball beginners and experts differ
—What results have you already achieved?
Kimura: We had expert and beginner baseball players experience our VR system. Their behavioral responses showed extremely interesting results.
First, fastballs, curves, and sliders were presented to them as strikes or balls. They were told to take a swing if they thought the pitch was a strike. The timing of the movements of their heads and necks and respiration were recorded, and we found clear differences between the experts and beginners.
Before the pitcher threw the ball, an expert began movements in anticipation of hitting the ball, such as synchronizing his foot stance to his estimation of the ball’s impact. The beginner, on the other hand, reacted once the ball was thrown. In short, the expert moves as he anticipates the pitch, whereas the beginner makes no preliminary movements and only moves after the ball is released.
The participants’ respiratory waveforms were also different. An expert took in breaths in sync with the timing of his swing. In short, he times his breathing with hitting the ball. No such behavior could be seen in the beginner.
—It takes less than a second for the ball to reach the batter?
Kimura: That’s right. It takes about only 0.5 seconds. It’s almost impossible to recognize the ball and hit it in a mere fraction of a second. This is why the expert player makes anticipatory movements from the time the pitcher winds up. What’s more, these are not conscious movements, but unconscious responses.
Figure 2. Behavioral responses to virtual baseball batting
We’ve studied the “broken escalator effect” before. When riding a broken escalator that doesn’t move, haven’t you had the experience of your body involuntarily pitching forward? When you ride a moving escalator, you need to lean in your body to get on smoothly and quickly. So for most people, their body reacts on its own when they step on a broken escalator, even though it is not moving. In the same way, even though participants know in their head that the environment is virtual, part of their bodies’ movements is unconscious response. We are making new discoveries by investigating this implicit brain processing.
—You have already observed differences between beginners and experienced players, right?
Kimura: Right. To put it another way, because a player is experienced, he already makes anticipatory movements. For example, when a soccer player feints, he has the skill to act in anticipation of his opponent’s movements. Such a skill involves learning variations of movements in soccer, and predicting what the next movement will be. This is why amateurs don’t have such a skill. They can’t predict the movements of opponents.
We are thus elucidating how responses differ depending on whether a person has sports experience and the level of the experience. Going forward, we plan to study the relationships between sports performance and how athletes apply their skills, the conditions under which they applied the skills, their gaze, mental state, and so on by combining our original data with new data like muscle activity data and eyeball movement data.
（Text revised: April 17, 2018）
（Japanese text written by Madoka Tainaka）
Sports Brain Science Project, NTT Communication Science Laboratories
Next: Article 9 (Part 2 of 2) Understanding Athletes’ “Skill” and “State of Mind” with Virtual Reality Technology
» Back to Articles
TopRELATED CONTENTS
SPECIAL CONTENTS | Feb. 18, 2016
Article 1 (Part 1 of 2) Exploring how to control the mind and body
Research on ‘mind’ and ‘technique’ as a means to improvement in sports
» READ MORE
SPECIAL CONTENTS | Oct. 19, 2016
Article 5 (Part 1 of 2) “hitoe”: Revolutionizing Sports and Medicine
The potential of functional materials able to get biological data, just by wearing them
» READ MORE
