Article 2 (Part 2 of 2) | Feb. 18, 2016
Understanding the inner mechanisms of the mind and body to find the essence of fine skills and methods for enhancing sports performance (Part 2)
Sensory feedback for conveying body movements through auditory and tactile perception
The Sports Brain Science Project aims to develop new methods to enhance athletic performance based on the collection and analysis of biometric data. What kinds of methods are being developed through the project? Will the day come when coaching can be provided through athlete’s clothing or special kinds of training rooms?
Sensory feedback for conveying body movements through auditory and tactile perception
—Your goal is to develop methods to actually enhance athletic performance based on the information you have gathered; can you give particular examples of such methods?
Kashino: Yes, the third step in our research is to develop methods to provide feedback to athletes based on the knowledge gained from the analysis of results in order to maximize their performance. The targets of the feedback, however, are implicit brain functions, which are part of the unconscious mind, so that no amount of head knowledge based on words or logic will help in any way.
For example, former professional and legendary baseball player and also manager Shigeo Nagashima, often used onomatopes such as “not boom, but biff!” when he teaches them how to bat. This is exactly what we would like to be able to explain to ordinary people (laughs). So, to overcome this challenge, we thought of the possibility of providing feedback in a way that will induce the body to intuitively and automatically react to stimuli. In other words, we want the feedback method to be also as implicit and intuitive as possible.
One of the methods we are developing to that end is “sensory feedback.” This would include, for example, feedback based on sound (auditory perception) that helps in terms of timing or rhythm, or feedback based on somatic perception, such as through touch.
—Since you have a long history of doing research on auditory perception, I’m sure the knowhow you have gained through those researches will be useful for this “sensory feedback” project.
Kashino: Yes. For example, in most sports, the key to winning is the quick coordination among multiple body parts. When you attach a sensor to each body part and generate a sound in response to the movements of the body parts during quick coordinated movements, the timing that sound is generated and the tone of the sound would vary among individuals. By providing those differences as feedback to the player in real time, you can let the player know intuitively about which part to move, how to move it, and when to move it. In this way, you can also present the tone difference between an amateur and a professional player, who have clearly different ways of moving their bodies.
Technically, we call this method sonification*1, which is an effective method because auditory perception has higher temporal resolution than visual perception, i.e., it is more sensitive to time. Rhythm and timing are important in sports, thus feedback based on auditory perception is very useful.
To begin with, sound significantly affects bodily control even before the person becomes aware of the perception. You have probably experienced unintentionally getting into step with the rhythm of a certain music you hear while walking on the street or unconsciously flinching upon hearing a loud sound. We want to use these attributes of auditory perception in providing feedback. Moreover, we would also like to skillfully incorporate the advantageous attributes of each of the five senses into the feedback mechanism, such as the superiority of visual perception in grasping shape or spatial position information.
*1 Sonification
Conversion and presentation of attributes and relationships of inputted data as sound
Enabling virtual experience a 160-km/h- fastball or a beanball
Kashino: Also, through the use of virtual reality or augmented reality*2, we would like to virtually create images of fastballs, breaking balls, beanballs, or other pitches that cannot be practiced in real life and simulate the experience for athletes.
For example, we are also conducting experiments to put off a batter by using a head-mounted display*3 to show a virtual image of a fastball coming directly to the batter’s face and look at how the heartbeat and muscle activity fluctuate in response. While some athletes would remain unperturbed, others would have their performance adversely affected by the stimulus; so, we want to look at what happens to the brain under such conditions. Or, we might just be able to create a way for athletes to experience being the batter for the ball that they pitched. That would be very useful in improving your batting skills by understanding how your pitch is seen by the other player.
—That sounds very interesting!
Kashino: The group of Senior Research Scientist Dan Mikami of the NTT Media Intelligence Laboratories has started creating that system, and received favorable responses from professional batters in regard to its usefulness. The impact of the beanball image was such that the athlete instinctively bent his body backwards, even though it was only a virtual ball. Going forward, we will conduct tests on what images are useful to further develop the system.
*2 Augmented reality
Technology for superimposing relevant computer-generated perceptual information, such as visual, auditory, tactile, and other sensory information, into the real-world environment.
*3 Head-mounted display
A compact display worn on the head like eyeglasses for displaying images at point-blank range from the eyeball. Due to the short distance from the eyeball, it functions like a large virtual display, and with the use of a half-silvered mirror, enables projecting virtual reality images right in front of the player.
Measuring biometric data using wearable devices
—You have shown us earlier a bit about how the measurements are made, but what particular sensors are used for measurement?
Kashino: An indispensible tool for measurement is a device called hitoe, which was jointly developed by NTT and Toray (Refer to No. 5). hitoe is a functional material that can be used to obtain biometric data, such as heart rate and electrocardiographic waveforms, simply by wearing it. Earlier, the subject wore this under his uniform. By wearing this device and taking measurements everyday, the athlete’s daily conditions, or his or her state of physical excitement during games can be monitored and analyzed.
We also use motion capture*4 to analyze movements. The suit worn by the subject earlier was for capturing motion based on inertial sensors rather than ordinary optical sensors. Both are very efficient sensors that enable taking measurements as soon as the subject wears them and making use of the raw measurement data for analysis.
Another important tool is the myoelectric sensor for measuring muscular activity. For this we use a wireless device so as not to hinder the body’s movements. Right now we are developing a system to enable measuring myoelectric activity using hitoe. Further, we also have a goggle-like device that allows measurement of eye movement and pupil diameter. Using these sensors, we plan to measure and accumulate a large amount of various types of data.
*4 Motion capture
Technology for digitally recording movement of humans or objects. 3D measurement of position and movement of joints and other regions that serve as focal points for movement (markers) enables recording “movement” as digital data suitable for computer analysis. This can be done optically by taking images of movements of the person wearing a body suit attached with markers or mechanically by recording movements through gyro (angular velocity) sensors or accelerometers attached to joints and other parts of the body.
Clothing and special training rooms that provide coaching: challenging the conventional wisdom in sports
—What are particular examples of results you hope to achieve?
Kashino: We can think of many different examples. Other than those that can be classified as multimedia training materials, going a step further, we hope to come up with clothing or special training rooms that provide coaching to athletes. For example, when an athlete wearing a smart suit enters the training room, his or her mental and physical conditions are instantaneously measured to provide information on his or her condition for the day and to display the optimum training menu suited to those conditions. Moreover, it would be great if we can create a system wherein clothing or special training rooms can function as competent “personal coaches” by providing accurate advice based on comparisons with past data or other people’s data.
—I think this can be used outside sports, too, like when you want to master a certain skill, this system can provide the most efficient training menu based on your current conditions and keep you motivated to train. What do you think would be the impact of your research on the world of sports?
Kashino: I think science and sports used to be two quite different worlds. From the field of sports, science was not something considered beneficial; while from the scientist’s point of view, sports has relied too much on experience and would benefit from a more systematic approach. We hope that we are somehow able to bridge the gap between these two worlds by presenting rational shortcuts in areas where systematic training was difficult to apply, such as in practicing special skills and in exercising mental control.
Also, there is more than one method for enhancing athletic performance, so I think it will be possible to provide the most effective training method suitable to each individual. For example, it would be appropriate for a type A person to be trained by a type A coach, but it would be disastrous for a type B person to train under a type A coach. If we can determine these specific types in detail through measurements, then I think we would be able to present effective training methods tailored to each individual.
Further, by presenting methods to enhance ways to move the body based on the person’s skill level, including children and top athletes, and even elderly persons, we hope to instill interest in sports among a greater number of people. I believe this would eventually lead to elevating the level of sports in general and contribute to the health and longevity of society as a whole.
（Text revised: April 10, 2018）
（Japanese text written by Madoka Tainaka）
Next: Article 3 (Part 1 of 2) Understanding the mental and physical state based on biometric data
» Back to Articles
TopRELATED CONTENTS
SPECIAL CONTENTS | Feb. 18, 2016
Article 2 (Part 1 of 2) Understanding the inner mechanisms of the mind and body to find the essence of fine skills and methods for enhancing sports performance
Using ICT to comprehensively understand mental and physical reactions
» READ MORE
SPECIAL CONTENTS | Oct. 19, 2016
Article 5 (Part 1 of 2) “hitoe”: Revolutionizing Sports and Medicine
The potential of functional materials able to get biological data, just by wearing them
» READ MORE
