Article 6 (Part 2 of 2) | Dec. 28, 2016
Improving Sports Technique using Sound and Rhythm (Part 2)
The potential of sonification systems
The Sports Brain Science Project is developing “sonification” systems that convert muscle motion into sound and provide it as feedback. It is also conducting research studying the relationship between body motion and sound more deeply, and whether it can be used effectively in sports training. Here, we speak with Dr. Mochida and Dr. Kashino about various approaches to sonification using the characteristics of sound, and prospects for the future.
Extracting the essence of using the body well and converting it to sound
—Are the effects of sonification systems on training are already being tested?
Mochida: Yes. It is still pretty limited, but there was a case where auditory feedback was given to a beginner learning to pitch (Kimura et al. 2014). In this case, rather than muscle activity, auditory feedback of acceleration in body movement was given. Beginner pitchers were asked to practice pitching with their non-dominant arm, and differences in movement before and after practice were studied.
Incidentally, “torso rotation” and “upper body forward inclination” were taken as key points for a good pitch. In other words, the torso must first rotate toward horizontal, and the upper body must fall forward. A good pitch involves linked movement in order from hips, to shoulders and arms. Beginners tend not to show such linked motion, but move them all together. We built that uses several accelerometers to measure acceleration in torso rotation and upper body inclination in real time, and generates wind-like sounds according to the magnitude of accelerations.
Participants were asked to practice using both auditory feedback and visual feedback. The acceleration pattern from an expert pitcher was converted to sound as auditory feedback and the corresponding video was used as visual feedback. Both were presented to participants as the model before starting to practice. They then practiced 100 pitches based on auditory and visual feedback, watching a replay of their throwing motion for each pitch. They were also asked to throw ten balls without feedback before and after practice, and motion capture was used to measure the motion of their arms, shoulders and wrists.
In comparing the motion captures, the pattern of motion progressing from hips to shoulders to wrist, was more prominent after practicing than before in both cases, of auditory and visual feedback. However, with the auditory feedback the participants were able to control their motion more precisely, and there was less variation in the timing of motion for each of the body parts (Figure).
Kashino: The essence of this experiment is that it focuses on two points: torso rotation, and upper body forward inclination. This was done because there are 500 different muscles in the entire body involved in throwing a ball, and if all of them were expressed as a sound, the listener could not make sense of them. As such “information compression” was needed. Essential information closely linked with the desired motion must be extracted, and conveyed back to the subject.
In such a short time, one can be aware of at most only one or two things. It would be impossible to imitate the motion of 500 muscles if instructed to move them in a particular way. What is important is to hit the vital points, and the other motion will follow accordingly.
But then, it can be extremely difficult to hit those vital points, and how to do so is a deep problem. Past research on speech has also confronted such issues, finding that even a single syllable, /a/, is different for each talker, and the shape of the mouth is different depending on the words preceding it. The motion changes when speaking quickly, or slowly, with emotion, or without. So why is it that listeners are able to recognize /a/ as /a/ in spite of this? It is not simply a matter of the spectral pattern; there are qualities that are extremely abstract. Finding these qualities is a nearly impossible task. Extracting the essence of good bodily motion is similarly extremely difficult. For this reason, we first attempt various approaches to more limited problems.
Pitching practice using non-dominant hand
Does prescribing out sound really work!?
Mochida: The sonification system also includes features to indicate excessive exertion. When we applied this to putting in golf, we made the sonification system to produce a sputtering sound when the player exerts too much force. The system can indicate that, the inexperienced players tend to constantly exert wasted effort, producing the sputtering sound, while experts do not waste effort.
Kashino: We are also focusing on the sounds produced as a consequence of a behavior, such as the sound of the swing, or the sound of the catcher catching the pitch. This is because when interviewing professional pitchers, they said that when they hear the sharp smack of the ball as it hits their catcher’s mitt, it lifts their spirits and actually improves their performance. A track athlete also said that when he feels bad, if he trains in a venue where he can hear the echoes of his footsteps, it improves his mood. These cases gave us the idea, and we are testing what can be achieved by prescribing such behavior-related sounds. We are attempting to support the movement by playing back the sounds louder than the actual sound, and adjusting the tone. Rather than simply feeding back the motion as it is, we prescribe an emphasized catching sound to see if we can prompt an improvement in performance.
—It’s true even for an orchestra, that performing in a hall with good resonance, produces a good, natural-sounding performance, doesn’t it?
Mochida: Professional performers must certainly be aware of this. I believe that the reverberation of the hall where they are performing, and sensing vibrations through touch, provides all kinds of information that helps in controlling body movement skillfully.
Kashino: If it’s a game, the cheers and yells of the spectators is another major factor affecting the performance of players. Sound really plays an important role in whether players are able to do their best when it counts. This is because sound has a direct effect on the operation of the autonomic nervous system, which controls levels of excitement and relaxation. This can also be utilized in training.
Using experience as a trumpet player in research
—By the way, is it true that you are trumpet performer Dr. Mochida?
Mochida: Yes, I started in the winds section in middle school. I started playing jazz in university and I still play in performances occasionally. With the trumpet and other brass instruments, when you put the mouthpiece to your lips and draw a breath, you have to control the shape of your mouth in various ways. It is not simply a matter of relaxing the lips for a low note and tightening them for a high note. The lips and tongue work to control tone in various ways. Of course, physical strength is required, but it is difficult to learn to apply that strength to the many muscles involved, and to control the balance.
Above all, the trumpet is an instrument with breadth of expression and able to show individuality in tone. For this reason, one must first decide what sort of sound to produce, and then practice it. Once you can produce the sound to some extent, then set your sights on the next sound and learn how to control your body for it. In that sense, I would say my experience performing with the trumpet has contributed to our current research as well.
—So are there similarities between how you breath and sports?
Mochida: Beginners must first practice and learn how to breath properly. Then, lung capacity is not too critical, but controlling the timing and volume of breath is very important to musical expression.
Incidentally, you have probably seen photographs of trumpet players on the covers of jazz magazines or on record jackets, contorting their faces as they play. Actually, when you see the top-level classical trumpet players, they can produce sound easily, hardly appearing as if they are playing at all. The moments of blowing hard are like the moments of effort when a baseball player throws a 150 km/h pitch, and if they played like that continuously, they wouldn’t be able to maintain it for even a minute (smiles).
—The sound from the sonification system that you played for us just before was different than a trumpet sound...
Mochida: Yes, that was just several pure tones superimposed, so it does not have much breadth of expression. It doesn’t have the richness of a violin or a trumpet. There is still a lot we do not know about how the qualities of sound can contribute to improving sports techniques. Rather than such qualities, we are still exploring timing and how to express the essential points, as we discussed earlier, by building various prototypes. In any case, we will be happy to contribute to development of sonification systems that can help control effort and timing intuitively, and even bring enjoyment similar that of playing a musical instrument.
（Text revised: April 16, 2018）
（Japanese text written by Madoka Tainaka）
NTT Communication Science Laboratories, Sports Brain Science Project
Next: Article 7 (Part 1 of 2) From Pro Baseball Pitcher to Researcher
» Back to Articles
TopRELATED CONTENTS
SPECIAL CONTENTS | Feb. 18, 2016
Article 2 (Part 2 of 2) Understanding the inner mechanisms of the mind and body to find the essence of fine skills and methods for enhancing sports performance
Sensory feedback for conveying body movements through auditory and tactile perception
» READ MORE
SPECIAL CONTENTS | May 30, 2016
Article 3 (Part 2 of 2) Understanding the mental and physical state based on biometric data
Leveraging the experience as a baseball player to tread into an unknown area of study
» READ MORE
