Sports Brain Science Project - SPECIAL CONTENTS - Article 6 (Part 1 of 2) Improving Sports Technique using Sound and Rhythm - Sonification: Converting muscle activity to sound -

SPECIAL CONTENTS Training the“Athletic Brain

Article 6 (Part 1 of 2) | Dec. 28, 2016

Improving Sports Technique using Sound and Rhythm (Part 1)

Sonification: Converting muscle activity to sound

What sort of effort will provide the most direct route to technique improvement in sport? Various approaches are taken, such as filming and reviewing one’s own movement, getting guidance from a trainer, and mental coaching, but there may be even better approaches. One possibility being investigated by the Sports Brain Science Project converts muscle effort and timing into sounds and provides them as feedback. This is being called “Sonification”. We spoke with Dr. Takemi Mochida and Dr. Makio Kashino, who are in charge of sonification systems.

Linking each body part to a sound conveys an image of the motion

—The Sports Brain Science Project is developing “sonification” systems, which convert muscle activity to sounds and provide them as feedback, with the goal of using these systems in sports training. Could you tell us more concretely what this involves?

Mochida: For these systems, sensors that measure myoelectric potential are attached to legs, arms and torso, and sounds are associated with muscle activity in each of these body parts. As an example, you could imagine a system that associates sounds with higher frequencies depending on height on the body, from legs, to torso, to arms. Thus each body part is represented by a sound of different pitch.

We are still investigating what sorts of sounds will best represent different movement conditions, but we expect that by even just listening to sounds corresponding to the activity of each muscle for a period of time, relations would begin to form within the brain, making it easier to understand the state of one’s body.

Kashino: In other words, if sensors are attached in various locations on the body so that, for example, exerting a calf muscle sounds a “Do” note, and using an upper arm sounds a “Mi” note, then by listening to this for some time, gradually the user will begin to understand what sounds will be produced for each effort, and be able to produce sounds themselves. The goal is that, with training, the relationships between sounds and the states of body parts becomes clear, making it easier to imagine bodily movement.

—Why did you focus on using sound?

Mochida: My original specialty is acoustics, so I’m less aware of sports than average, but my research colleague, Dr.Toshitaka Kimura, whose research specialty body movement, was talking about how he always thought it would be interesting to convert body movement to sound. Before joining this project, I began really wondering whether sound could be applied to sports training. Now, I have been working on it seriously for almost a year.

Humans use sound for communication in the first place, moving our mouths to control the sound and exchange information. We know how to move our mouths to produce a particular sound. Thus, it seems that each person has a sort of auditory image that corresponds to movement. For example, upon hearing an “oo” sound, we might recall an action of pursing the lips.

On the other hand, when engaged in sports, the coach will often use words to teach the subtleties of how to move. We often hear phrases like “ease up a bit,” “swing your arms,” or “put your hips into it.” However, the image of a motion formed when hearing a verbal description can be completely different depending on the person. If there is potential to convey the wrong image in words, perhaps the characteristics of motion could be conveyed more effectively using sound, which can express power, tempo and rhythm. It may be possible to convey the subtleties of motion more accurately using sound images, and in turn, use it to improve body positioning.

Motion cannot be imitated, even by watching and copying

—Expressing how a body is used with sound sure is an interesting idea.

Kashino: I think so too. When trying to understand movement in sports, a common method used is motion capture. With motion capture, markers are attached to locations on the body and the motion of these markers is traced. This captures the form of the motion, but it is still very difficult to know what effort to exert where, and with what timing. This can often lead to mistaking what is the cause and what is the effect.

For example, when throwing a ball you may be instructed to “swing your arm”, but rather than swinging your arm, you may need to rotate the lower part of your body, with the result that your arm swings around. Thus, even if you pick out the local arm movement and copy it, you may not be achieving the motion you desire.

Photographic playback and slow motion video are used to research the form of pro pitchers, but a common mistake in doing so is to mimic how the arm twists and palm faces upward when pitching straight. However, the pitchers are not intentionally turning their palms to face upward; it is a result of swinging the wrist downward, so the hand naturally ends that way. Regardless, an amateur seeing video of a pro pitcher and copying it will put effort in the wrong place. Trying to reproduce the form with compensating movement in this way can result in using the wrong muscles, and even cause injury.

So, why is sound a good option? Sound has better time resolution than vision. In other words, it has good sensitivity to timing. It is also able to express conditions of exertion and motion easily. If you strike a drum strongly, a loud sound is produced, and softly, a quiet sound is produced, so there is a strong connection between power and sound. It is also easy to draw correlations between actions such as rubbing, hitting or stroking, and qualities like weight or hardness. We believe that if we can use the information naturally present in sound and the abilities of our sense of hearing to distinguish them, we should be able to communicate motion more easily.

Audible comparison of how a professional and an amateur use their bodies

professional

amateur

—You have played results from the sonification system tests expressing muscle activity in various parts of the body while throwing a ball, and we can hear differences in the length and rhythm of the sound between the former pro baseball player and the amateur.

Mochida: Yes, with the amateur, each of the sounds is longer, with effort being applied throughout, but the muscle activity of the pro turns on and off clearly, and each sound can be heard clearly and rhythmically.

Kashino: You can also hear that with the pro, there are moments during the motion where the pro relaxes significantly. When shifting his weight from the pivot leg to the front leg, there is a moment when he is totally relaxed. This is evidence that he is using the slope of the mound to shift his center of gravity, not wasting any energy. Fans of sandlot baseball like me can see that we are trying too hard, putting in too much effort all the way through. Such differences in how the body is used are expressed in differences in the length and rhythm of the sound patterns.

However, as Dr. Mochida just said, we still need to study how sound should be associated with the body, what mappings would be the best, and how to express various aspects of movement.

Mochida: That’s right. We will need to verify many things such as what representation in sound would be good to help increase the speed of a pitch. We are actually starting to conduct experiments with several prototypes. However, body image differs from person to person, and it may be that the optimal sounds also differ from person to person.

—Have you actually changed anything from hearing the sound of your own muscle activity?

Kashino: Yes. Although I wasn’t easily able to achieve increases of 10 km/h or more in the speed of my pitches when I tried it, I did feel that it was very helpful in learning to throw more comfortably at the same speed. I was amazed at how easily and effortlessly I could throw the ball at about the same speedy, just by knowing the timing and level of effort to apply.

Learning to use the body well, by playing it like a musical instrument

Kashino: Incidentally, before throwing the ball, I compared the sound I was producing with that of the pro and picked out differences in the length and rhythm of the sound, using expressions like, “there’s a rest here,” and “this part is staccato.” Then, when I tried to reproduce the pro’s crisp, staccato sound with my throw, I started to get the knack of how to throw more easily. In this way, I began to see my body like a musical instrument and feel like I was playing it.

—Isn’t it true that you can do this because you have strong sensibilities with respect to sound? I play violin as a hobby, and know that making using shorter bow strokes and relaxing creates better and clearer deep sound, but I always feel this is difficult to achieve.

Kashino: It’s true that sensitivity to sound varies considerably from person to person, and the way that it is embodied may also vary. For myself, I’ve always had an audible image in my head when I move my body, so sound and movement are very closely connected for me. For example, when I’m catching a grounder in baseball, I have an image of moving my body to a rhythm. So I have a sense of what Shigeo Nagashima (Japanese legendary baseball player/coach) means when he says things like “not BOOM, but Pyew!” Of course, there are people who will find it more difficult to get such an impression from sound, so figuring out how to adjust for those situations will also be a significant issue.

Mochida: It appears that ability to express using sound may also be related to proficiency. With the violin, the more experience you accumulate, the more you can hear the difference in sounds and the easier it is to produce the right sound. Similarly, the more I learn the correspondence between the sound and how I am using my body, the better I will be able to manipulate my body. That’s why it is similar to how we use tools as an extension to our own bodies, as with a glove, a bat, or a musical instrument.

Similar to how some people get extremely proficient at musical instruments like the violin, even as elementary students, producing a good sound is more related to learning how to control strength rather than the amount of physical strength. It is about how the physical motion of moving the bow is associated with the sound produced and how this can be controlled.

Kashino: When pitching, we tend to be conscious of the form and position of our elbows or legs and to try to control them, but as former pro baseball player, Masumi Kuwata would often say, “its better not to get caught up in the form.” Instead, he would say to put most of your weight on the hip joint of your pivot leg at the beginning, and then load it onto the hip joint of your forward leg, as though you are throwing a large basin. Then, if you are a right pitcher, imagine you have an eye on your left shoulder blade, and try to throw the ball as though that eye is looking at the catcher. You can forget about the motion of other parts, like elbows and arms.

The point is that underlying the form there are critical moments of effort, and the timing when effort is applied is crucial. It is extremely important in sports to make many body parts work together with agility. This should be easier to express with sound, and its high temporal resolution than by looking visually at the form. Of course it won’t be easy, but we hope to develop revolutionary training methods that will be totally new.

（Text revised: April 16, 2018）

（Japanese text written by Madoka Tainaka）

Takemi Mochida, Ph.D.
NTT Communication Science Laboratories, Sports Brain Science Project

Next: Article 6 (Part 2 of 2) Improving Sports Technique using Sound and Rhythm
» Back to Articles

Top

Improving Sports Technique using Sound and Rhythm (Part 1) Sonification: Converting muscle activity to sound