Sports Brain Science Project - SPECIAL CONTENTS - Article 8 (Part 1 of 2) DIGITAL CONTENT EXPO 2016 Symposium “The Possibilities of Content Technology in the Olympics and Paralympics - Can we strengthen humans through the power of technology? -

SPECIAL CONTENTS Training the“Athletic Brain

Article 8 (Part 1 of 2) | Mar. 14, 2017

DIGITAL CONTENT EXPO 2016 Symposium “The Possibilities of Content Technology in the Olympics and Paralympics” (Part 1)

—Tokyo 2020 Official Program (Economics/Technology) / Oct. 27, 2016 at the National Museum of Emerging Science and Innovation (Miraikan) —

Can we strengthen humans through the power of technology?

In recent years, by utilizing cutting-edge technologies such as drones, head-mounted displays (HMD), and big data analysis, it is possible to enjoy watching sports from unprecedented viewpoints and to strengthen humans’ physical abilities. Particularly with the coming 2020 Tokyo Olympics and Paralympics, expectations are high for advanced technology that contributes to sports. Researchers and athletes gathered at the DIGITAL CONTENT EXPO 2016 Symposium held at the National Museum of Emerging Science and Innovation (Miraikan) on Oct. 27, 2016, to discuss the possibilities and expectations of advanced technologies.

Speakers

Masahiko Inami,　Professor, Research Center for Advanced Science and Technology, the University of Tokyo
Makio Kashino, NTT Fellow, NTT Communication Science Laboratories; then-Executive Manager, Human Information Science Laboratory / Project Manager, NTT Sports Brain Science Project (current)
Koji Murofushi,　Sports Chief (and Sports Director), Organising Committee of the Tokyo Olympic and Paralympic Games
Kuniko Obinata, Vice Chairman, Japan Paralympic Association / Steering committee member, Japan Paralympic Committee / Member, Industrial Structure Council’s 2020 Future Development Group / Gold Medalist, 1998 Nagano Paralympic Games and 2006 Turin Winter Games / Dentsu Public Relations Operations Department / Senior Consultant, Olympics/Paralympics Division

Expanding Human Capabilities and Aiming to Cultivate New Sports

Moderator: Now we will begin the DIGITAL CONTENT EXPO 2016 Symposium, “The Possibilities of Content Technology in the Olympics and Paralympics —Tokyo 2020 Official Program (Economics/Technology)—.” First, we will have a presentation from Dr. Masahiko Inami, Professor at the University of Tokyo’s Research Center for Advanced Science and Technology.

Inami: I will introduce the possibilities of content technology related to watching experiential sports. Before that, let me briefly introduce myself. I am currently a professor at the Research Center for Advanced Science and Technology at the University of Tokyo. The theme of my research is building a system to support humans to have freedom of action and expression. I work in areas such as human systems understanding and human augmentation engineering.

Additionally, I am co-founder of a group called the Superhuman Sports Academy. The theme of this association is to reinvent sports with modern technology. In other words, we aim to create new sports by expanding human capabilities with cutting-edge technology and acquiring the ability to exceed human physical abilities. We are also promoting research and development of ways for all people to enjoy sports vicariously, whether as competitors or as spectators. More than 50 researchers and cultural figures are involved in the group, including co-founders Prof. Junichi Rekimoto of the University of Tokyo and Prof. Ichiya Nakamura of Keio University.

Why “superhuman”? When I was little, I thought I wanted to fly. So I tried various things, such as training my body and jumping from high places, but one time, I injured myself falling from a tree. I became disappointed in my lack of athletic ability and gave up on being an athlete. Instead, I thought about improving the human body’s ability and senses by using technology. I’m now researching technology to expand human capabilities by developing the field of “physical informatics.”

In particular, we are focused on research about the social activities and sports participation of seniors and people with disabilities. Also, at the University of Tokyo, to which I belong, we began the “University of Tokyo Sports Science Initiative” in 2016 across the university, and I joined as a steering committee member. Further, at the National Museum of Emerging Science and Innovation, where we are today, we have various activities related to the body and sports. These include standing up the “Living Lab Tokyo Project” and the “Superhuman Sports Project,” with which you can actually experience advanced technologies.

One such research is JINS MEME, a glasses-type device that can measure the movement of the eyeball that we developed with JINS, a glasses maker. Although they look like ordinary glasses, they actually have electrodes on the bridge. They can detect blinking and eyeball movement by measuring the eyeballs’ electrooculography. For example, if you play sports wearing this glasses-type device, you can capture the movement of the eyes and head or the field of vision, etc. I think that this will allow us to clarify the correlation between eye movement and performance.

Next, with regard to technology that expands human capabilities, we introduced various things in Birth of the Superhuman: Humans Beyond Science Fiction, published in 2016 (NHK Shuppan Shinsho). As one recent, fascinating technology, researchers at Arizona State University in the United States have announced a technology that allows a person to run fast by strapping on a jet engine. (Note that it seems difficult to stop safely!) Another unique system that has been developed allows one to shoulder a device called a jet pack and fly over the surface of water. I actually tried it, and my childhood dream of flying in the sky finally came true!

Our challenge is to make these advanced technologies useful for the Olympic and Paralympic Games. Utilizing the University of Tokyo’s Research Center for Advanced Science and Technology and cooperating with the members of the Superhuman Sports Academy, we are initiating various activities, including planning events that combine technology, skills, and culture. We are also accelerating interdisciplinary approaches. Last year, we held a research meeting at the University of Tsukuba and also worked on the development of a suit to assist the elderly and others. Thanks to these efforts, we are also beginning to attract attention from the media, including recent coverage of our activities in the sports magazine Number.

Technology changes sports spectatorship and training

Inami: Well, I think there are various possibilities for today’s theme of “sports and content technology.” For example, in this autumn’s Japan basketball league opening game, the court covered with LED light panels became a hot topic. This was a collaboration of Nike, Rhizomatiks, and AKQA, Inc. (UK). It made not only the opening ceremony exciting, but it can also be used according to developments in the game such as points, and is expected to make sports spectatorship even more engaging.

Also, recently, projection mapping that projects computer graphics images onto buildings and objects has been used in various settings, but from now on, it is said that these technologies will help make it easier to enhance night game spectatorship.

We are also paying attention to the research on force tactile devices by Keio University Graduate School of Media Design’s Associate Professor Kouta Minamizawa. For example, he is trying to record and reproduce the shock transmitted to the hand when hitting a ball with a racquet. As you know, when in tennis or golf you hit a ball with the racquet or driver, the force and sensation you feel in your hand will change depending on whether you hit it with the center or the edge. In addition to reproducing this difference in force tactile sense, he shows video that matches the movement. Application of this technology holds great promise for being able to easily enjoy a realistic experience of playing sports from home, not only with video or sound, but also with force tactile sense.

Also, I believe that such content technology can be utilized not only for enjoyment, but also for sports training. Professor Jun Rekimoto of the University of Tokyo has been conducting research to help improve form by having a drone follow an athlete who is running or swimming and checking the image in real time. Various types of drones are now available on the market, and in the near future, you may see something like this – people training while checking the image with a smartphone. In this manner, advanced technology is beginning to greatly change the very ways of sports training.

Research using big data analysis that contributes to sports training is also being actively conducted, mainly in the United States. In the field of basketball, recently research results were announced mapping the probability of making a shot based on from where in the court the ball was thrown. They found that it is easiest to make a shot from a 45-degree angle with respect to the basket, and furthermore, that is best to aim for 1-2 inches past the center of the 18-inch rim, that is, 11 inches past the front of the rim. Big data analysis is uncovering new training methods, including training devices that have been developed based on these results.

Also, Artificial Intelligence (AI) is attracting more attention. This year, a hot topic was Google’s AlphaGo beating the Go world champion. Young Go players are learning new techniques based on AI’s moves and polishing skills that never existed before. This will happen in the sports world as well. For example, “Hitachi AI Technology/H” developed at Hitachi is AI developed to maximize given targets such as sales and work efficiency. When this AI was connected to a robot that could use a swing and made to autonomously learn the movement through reinforcement learning, interesting results were obtained. The robot could only move randomly and clumsily at first, but gradually got the hang of it and learned how to move to maximize its swing. Moreover, it discovered a way to move its knees better than humans, by bending and stretching its knees both forward and backward. In this way, by utilizing AI, we may be able to discover things such as efficient ball throwing and hitting methods.

Expectations for virtual reality are also increasing. The University of Tsukuba’s Team Champignon’s “Childhood,” which won the overall victory at the International Collegiate Virtual Reality Contest sponsored by the Virtual Reality Society of Japan in 2014, allows the user to see images in goggles on the head of images from a camera attached to the waist. This gives you the strange experience of your line of sight being at your hips. When you are talking to the university students, you unintentionally look up. In other words, you recall the bodily sensation of being a child. If you can apply these techniques and see the landscape of players who are taller than yourself, you may be able to create a strategy of attack. Also, by combining this with the device introduced earlier that can track one’s line of sight, if you can clarify what the other party is looking at, this will be useful for establishing a strategy accordingly.

In the future, I think that advanced technology will greatly change the ways of watching sports. Prof. Rekimoto has developed a way to acquire and synthesize real-time images of 360-degree omnidirectional viewing from a person wearing a small camera around the crown of the head. With this technology, you can see images from the point of view of the athlete doing giant slalom skiing or spinning during figure skating. It may be difficult to actually put this on the head of the athlete during the Olympic Games, but if you put it on a referee or a person near the field and send the video to 4K or 8K TV, wouldn’t it be possible to have a spectacular viewing experience while you are in the middle of the living room, as if you were at the venue?

The possibility of new sports through the fusion of sports and technology

This year, in Switzerland, an international convention called “Cybathlon” was held for athletes with disabilities who used state-of-the-art robot engineering and biomechanics technology to extend their abilities. This is an international competition led by Prof. Robert Liner of Zurich Polytechnic University and is consistent with the direction that we are aiming for in superhuman sports. Of course, we are aware that “technology doping” is being raised as a serious problem, but nevertheless, I think it is a very interesting attempt to positively exploit advanced technologies and explore new ways of sports.

In the Cybathlon, besides a competition to grab and carry a fine item using a robot prosthesis, there are various events such as operating a computer avatar using brain waves and competing on speed, or an obstacle race using an electric wheelchair, etc. I watched the games and had been worried that it would be lacking in excitement. But both the spectators and the players were very enthusiastic. It was a very interesting event suggesting one way of fusing technology and sports.

Moderator: Prof. Inami, thank you very much. Next, we will hear from Dr. Makio Kashino, Senior Distinguished Researcher and Executive Manager of the Human Information Science Laboratory at NTT Communication Science Laboratories. He will give a presentation entitled “Athletes’ Brains: Revealing Mechanisms and Applicability to Training.”

The intimate relationship between sports and the brain

Today I will speak on the theme of the athlete’s brain. I originally researched the human sensory systems, primarily the sense of hearing. Examples include research on what the brain does to listen to sounds it wants to hear in a noisy environment, reading the emotions listeners feel from sounds they are listening to based on their eye movement, and exploring the characteristics of the sensory systems of people with autistic spectrum disorder who have difficulty communicating with people. In doing this, I recently realized that sports is a very interesting topic for advancing our research, and last year started a research field called “sports brain science.”

Sports and the brain may seem somewhat distant at first glance. In fact, however, the brain has a very important role in sports. Recently, I had an interview with the magazine Number and had the opportunity to speak with boxer Naoya Inoue. He is the owner of a powerful punch, and is known for his rapid response. Thus, it is said that he has never received a punch that has caused blood to flow. He is a rare player who can, for example, immediately change hands in response to the opponent and deliver punches.

So what about him is so much better? Normally, it may be thought that superior athletes have extraordinary muscular power, that they have dynamic visual acuity, or that they have great reflexes. However, when Inoue’s physical ability was examined for a television program, it turned out that he is average as an athlete. So I asked him, “What do you think that you’re the best at?” He responded, “spatial recognition ability.” Inoue said that spatial recognition ability is not being good at directions, but rather when the first round starts, to quickly grasp how close you can step toward the opponent, or the distance from the other party. Also, Inoue is good at throwing a quick counterpunch, but I was surprised to learn that that is not because of his fast response, but because he planned it from the previous movement and his opponent moved as he anticipated.

Furthermore, when I asked, “What do you watch during a match?” he said, “I am looking around the opponent’s eyes, but I am not looking at his eyes. I can’t see his feet, but sense his whole body’s movement with my entire field of vision. This is precisely Musashi Miyamoto’s “perception.” In Miyamoto’s The Book of Five Rings (1645), there is a famous phrase, “make perception strong and sight weak” to describe the gaze of the art of war. Here, “perception" is not seeing shape through visual information, but the attitude of viewing while seeing behind shape to anticipate the other’s intentions and predict what will happen next. In brain science terms, it can be said that you hold a model of the opponent’s movement and, based on this, interpret visual information. By giving meaning to what you saw and interpreting it, it is for the first time possible to respond appropriately according to the situation.

In other words, while it is not that he is moving while thinking of each little thing, it looks as though he is acting reflexively without thinking about anything. Yet in reality, his brain is fully operating. We call this reflexive brain activity “implicit brain function.” implicit brain function is unconscious function that can’t be controlled consciously but is a function of the brain that plays an important role in instant judgment that is indispensable to sports. However, this implicit brain function is not limited to sports and plays a very important role in our casual behavior, such actions or decision-making in daily life.

“implicit brain function” responsible for rapid reaction

Kashino: The other day, as part of my research, I measured the top Japanese junior high school women’s softball players. For women’s softball, the distance from the pitcher to home base is about 13m. At that distance, they throw the ball at a speed of 90 km/h. When converting this into baseball distance, this is around 125-135km/h. Japanese national team member Yukiko Ueno throws in the 120 km/h range, so if you convert this to baseball, it is 150-160km/h, which is equivalent to Shohei Otani. The time it takes for the ball to get from the pitcher’s hand and reach home base is only 0.4-0.5 seconds. It is nearly impossible to do something consciously in such little time. In the first place, it takes about 0.2 to 0.3 seconds just to obtain information from the eyes and process it in the brain, and it takes about the same amount time to move from there in some way. In other words, there will never be enough time for you to move after capturing that information. But in fact, the body is reacting unconsciously so you can hit the ball. It is the implicit brain function that is responsible for this movement.

So, about such reactions, if we look from the brain’s point of view, we can get closer to the mystery. The visual cortex of the brain is located in the occipital lobe of the back of the head. In fact, there are multiple pathways in processing. Roughly divided, there is a pathway that passes from the occipital lobe to the upper side of the head and the parietal lobe, and a pathway that passes through the lower side of the head. The pathway through the parietal lobe is the one that captures movement and moves the body. Meanwhile, the lower pathway is the one that recognizes colors and shapes. Stated differently, seeing characters and recognizing the human face goes through the lower pathway, and the reaction to a flying ball uses the upper pathway.

Although the pathway for seeing colors and shapes is able to get detailed perception, processing is slow. The pathway for seeing motion can only obtain coarse information, but processing is very fast. In other words, information passing through the parietal lobe does not rise to consciousness, but it is responsible for rapid response. So, if you were to train yourself to react quickly to a ball, it would be a better strategy to train the upper pathway.

Still, there may be some people who think this is strange. This is because you have the awareness that you hit the ball. Although you may think you are consciously hitting the ball, in fact, consciousness is something that only emerges about 0.5 seconds after, and is the retroactive creation of coherence. Thus, there is a considerable gap between what you’re conscious of and the events that are actually occurring. If you consider this, you will find that strengthening dynamic vision, such as reading words presented for just a moment, is not very useful. To improve sports skills, it is necessary to efficiently train the brain’s ability to control quick reactions.

Exploring what is happening in the brain and the body to help improve performance

Kashino: Well, based on these things, the keyword of sports brain science is “implicit brain function,” which is movement without being conscious of it. The first step is to capture what is happening in the body and the brain. For that reason, we are starting measurements mainly through wearable sensors so as not to disturb athletes’ performance. For example, NTT is developing a conductive functional material, hitoe®, in collaboration with Toray. Sportswear incorporating hitoe® is used to acquire heartrate data. Furthermore, although it is in the research stage, we are also using hitoe® to measure myoelectricity, i.e., the movement of muscles. Also, we are working on obtaining various biological information, such as examining where the eyes are looking and the condition of the pupil at that time, and the movement of the body in real time.

Next, we use machine learning and signal processing techniques to analyze the biological information thus obtained and try to discern what is essential among it. Furthermore, the ultimate goal of this research is to feed back the research results to the athlete. So how do you provide feedback to the athlete? For example, we think that we can make it so that the body moves instinctively, or so that the brain works of its own accord. One method is called “sonification,” which converts muscle activity into sound and conveys it. For instance, by expressing the difference between the professional pitcher and the amateur’s movement via conversion to sound, it is possible to clarify the difference in throwing rhythm or the degree of power input, etc. We are seeing if this is useful for training.

Another thing that we utilize is virtual reality (VR). Using a head-mounted display (HMD), you can reproduce the ball being thrown by the pitcher and experience the feeling as if you were at the plate. For example, how it would feel if Shohei Otani throws a straight fastball of 165 km/h, and then throws a forkball of 150km/h, can be in principle reproduced. Or, the response of the batter’s body with a hit-by-pitch flying towards the face followed by a curve ball can be measured in real time thanks to VR.

In addition, we are also taking measurements in the real word. We are using hitoe® and other sensors to measure biological information during games. Actually, I am not that good at but love baseball, and am a pitcher for Masumi Kuwata’s grass-lot baseball team. The other players are former professional baseball players or former high school baseball stars, so there is considerable pressure in the game. As a result of the players measuring themselves during the game, we learned that their heart rate was surprisingly high due to the tension (see Article 3 for details). Although the game provided about the same activity intensity as their daily exercise, during critical points in the game, their heartrate increased by over 60 beats per minute. Moreover, even after one’s turn at the mound, at points where one’s teammate had the opportunity to reverse the score and win the game if they made a hit, heartrates rose by 40-50bpm. With the development of a wearable sensor, it has become possible to record in detail such mental and physical changes occurring in the real world.

As you have realized from my discussion so far, to win in sports, it is not enough to simply increase muscular strength, cardiopulmonary function, or stamina. Successfully controlling one’s physical and mental condition is essential when the game counts. The brain is the very thing that holds this key. However, it is hard to be conscious of implicit brain function and it is difficult to control. Therefore, we are trying to clarify those workings with the cooperation of top athletes who likely have outstanding implicit brain function. We are conducting research together with Dr. Koji Murofushi, who will also speak today after this.

In the future, we will acquire even more varied types of biometric information in such a way that it will not interfere with performance. Using advanced technologies including artificial intelligence, such as deep learning, we will accelerate clarification of the brain and develop new, revolutionary training methods.

Nevertheless, in a real environment, it is not easy to elaborate causes and results because all factors are involved in a complex environment. Still, I think it will become possible to create a new methodology based on scientific evidence by continuing to acquire and accumulate data in a real environment. Furthermore, since each person, body, and brain are different, we would like to devise a training method specialized for that person. If we can do that, I think we can also contribute to reducing tragedies such as damaging the body through excessive training or failing to reach one’s full potential under an incompatible coach.

Moderator: Dr. Kashino, thank you very much.

（Text revised: April 17, 2018）

（Japanese text written by Madoka Tainaka）

Next: Article 8 (Part 2 of 2) DIGITAL CONTENT EXPO 2016 Symposium “The Possibilities of Content Technology in the Olympics and Paralympics”
» Back to Articles

Top