We explore an underleveraged concept in the gaming industry - Synesthesia - and delve into the career of one of its strongest advocates: Tetsuya Mizuguchi
Before diving into our second article, exciting news! We've added a newsletter feature. Stay in the loop by signing up to receive notifications when we publish new articles (no spam, promised).
If this is your first time here, check out our About Us page! We have also covered Omori, the successful indie game from OMOCAT, in a recent article.
Our inspiration for this article came from an unexpected source - Italian traditions. My younger brother gifted me Humanity on PS5 for my "onomastico" (something that Google translates as "the feast of the saint you are named after"). Humanity’s producer, Tetsuya Mizuguchi, is famous for his unique and recognizable style, but we believe that many people are unaware of how and why he has been so successful in the industry. You might have played games such as Rez, Tetris Effect, Lumines, or Child of Eden without knowing that all these games are his creatures.
For this article, we decided to play all of Mizuguchi's main games, exploring his vision and how he shaped his career based on the fascinating concept of synesthesia.
Jump to a specific chapter here:
Tetsuya Mizuguchi - Early career
Before discussing his games, let's briefly explore Mizuguchi's early career.
As mentioned in his talk at GDC 2016, one of the most impactful events in his early career was the attendance of the Street Parade - the World's largest electronic music festival - in Zurich, Switzerland. Mizuguchi graduated in Japan with a major in what can be translated as Media Aesthetics at the Nihon University College of Art, where he became familiar for the first time with the term “synesthesia”. Anyhow, it wasn't until Street Parade that he realized what the term really meant to him. He describes it as: “Sharing the physical space with 10 thousand other people, all moving together, with loud music and screen reflecting colors all around”. It was an enlightening moment for him.
From there, his search for synesthesia in video games began. Mizuguchi started his gaming career at Sega around 1994, working on Sega Rally first and then on a music-related game in 1999, Space Channel 5 (which is apparently one of the biggest inspirations for Alex Rigopulos, creator of the Guitar Hero series). Still while at Sega, Mizuguchi produced his first game, Rez, in 2001. This is his first attempt at leveraging synesthesia in gaming.
What were Mizuguchi's inspirations? When talking about synesthesia in gaming, Mizuguchi uses Xevious (1982) and Xenon 2 - Megablast (1989) as examples. These games are classical space shooting arcade games, clearly built upon the famous Space Invaders (1978), but they have a peculiar characteristic: sounds from shooting enemy’s spaceships and from their explosions are perfectly matched to the soundtrack. These games made Mizuguchi feel as if, while playing, he was creating music.
What is Synesthesia?
Synesthesia is defined as a neurological phenomenon that occurs when stimulations coming from a sensory or cognitive pathway lead to an automatic and involuntary experience with a second pathway. An indication of synesthesia is for example when a perceived stimulus (such as sound) causes a clear reaction of another sense (for example sight). There are many kinds of synesthesia, albeit it is said that very few people are affected by it and it is very hard (if not impossible) to scientifically prove it.
If you are a classical music enthusiast like me, Scriabin’s Prometheus: Poem of Fire is a synesthetic classical symphony, which employs the so-called color organ as one of the instruments, and the live performance usually includes the projection of colors based on the circle of fifths all around the hall.
A more common, and easy-to-understand example, is when a person affected by synesthesia “sees” colors when looking at numbers and/or words, like the image below.
We believe Mizuguchi is the first in the gaming industry to have built a career on synesthesia: he has been able to make players “feel” sound and colors when smashing buttons on a joypad.
In the following chapters, we explore Mizuguchi's key games, highlighting how synesthesia is applied in each.
Rez (2001)
Rez, released in 2001, is an “infinite-runner” action shooter game.
The game's aesthetics, a mix of appealing low-poly visuals and a dark yet contrasting world, coupled with a continuous, pulsating soundtrack, create an immersive experience.
The game is short, you need around 2 hours to complete the main story, but provides an entertaining experience.
The synopsis is interesting: the world of Rez is set in the near future on a massively overpopulated Earth with a society crippled by crime rates, and on the verge of meltdown. In an attempt to resolve this chaos, a new global network, Project-K, was created to oversee a self-aware artificial intelligence named Eden. However, the increasing information flow confused Eden, allowing her to begin questioning the meaning of its existence and the consequences of her actions. Confusion took over, until she shut herself down.
The mechanics are simple, easy to learn, and straightforward. Your goal is to guide a digital hacker through different "worlds" while defeating IT-inspired enemies such as firewalls and viruses to finally free Eden again. The player has only two attacks available:
The Lock-on attack allows you to lock on multiple (up to 8) targets at the same time and then hit them simultaneously.
The Overdrive sends projectiles in all directions to all the on-screen targets.
As you may understand, the gameplay is pretty basic. But what is really cool is how Mizuguchi has perfectly aligned shooting, ambiance, and damage sounds with the soundtrack to create a coherent and never overwhelming flow that keeps the player fully immersed in the game.
Rez and Synesthesia
Interestingly enough, Mizuguchi and his team followed a “standard” game development process to develop Rez, but they built it based on some unique game pillars:
They developed a "normal" rail shooter game and then applied some new concepts to it.
To make this work, Mizuguchi introduced what he calls Quantization. This process is a way for developers to force the synching of random rhythm inputs (due to player actions, i.e., pressing buttons) to the background music, trying to reach something that would feel good to the human ear.
The result is great: in Rez, visuals and sounds are perfectly connected and they enhance each other, delivering the intended goal of letting players see sounds and listen to images.
There are a lot of examples in the game that show this. We want to show a couple of them:
Listen how the shooting sounds are perfectly aligned with the background music!
The character pulses rhythmically with the background music the whole time
What we think is really cool (and a rare accomplishment so far in the gaming space) is that some sections of the game are full of elements to shoot, the music is really loud, and the low-poly effect is all over the screen, but we never felt that the impact was too chaotic or annoying.
We played the whole game in one session, and, after finishing it, we felt like leaving Rez’s world and going back to the real one.
Rez is the first attempt of Mizuguchi at implementing synesthesia in a video game, but it is also clear that the game has been heavily limited by the technology available in the late '90s. Mizuguchi has reworked on Rez many times over the following 20 years, leveraging new technologies and creating some really interesting concepts that go far beyond games.
Rez Infinite - Rez VR version - was launched in 2016, but what we think is more interesting is the so-called Synesthesia Suit. This is a real suit that can be worn by the player to play Rez, with 26 vibrators that transfer the vibrations all over the body according to the rhythm of the sounds played in the game. Moreover, there is an additional layer of the suit that lightens up selected vibrators rhythmically, creating an incredible visual effect. The Synesthesia Suit, showcased at the Media Ambition Tokyo event in 2016, might not be something you use every day to play video games, but we would love to try it at least once to see how it feels. Check out the video!
Rez Legacy
Before proceeding to the next game, we want to mention that synesthesia today is still a niche feature, but some recent games have been influenced by Mizuguchi's approach.
We will not go into the details, but we wanted to mention a couple of examples.
A recent case is Hi-fi Rush, which is based on the idea of visuals and gameplay connected to the rhythm of the music. For readers who have not played it, the gameplay is based on the player having to press the buttons rhythmically to execute combat actions such as dodging and attacking. Additionally if you pay attention, you can see how the UI and many on-screen elements pulse or move alongside the beats of the music (see for example the fences in the GIF below), which is pretty similar to what happens in Rez.
Also, the Guitar Hero series (we mentioned before than one of the founder was inspired by Mizuguchi's Space Channel 5) has a couple of UI and in-game features that are inspired by Rez and leverage synesthetic effects. When playing the right note on time, a small fire is shown on screen and, when activating the "Star Power", the screen becomes light blue and the player characters start doing crazy actions.
Lumines (2004)
Lumines is the second main game from Mizuguchi. It is a clever application of the synesthesia concept to a known pattern since the game is clearly inspired by Tetris.
The objective of the game is to survive as long as possible while placing 2x2 bi-colored blocks in a 16x10 grid. The blocks can be rotated before being placed, allowing the player to form 2x2 squares of a single color which will be erased when the Time Line passes over them. A GIF is clearer than a thousand words.
Lumines and synesthesia
As in Rez, Lumines gives a lot of attention to the connection of visuals, gameplay, and sound.
When playing the Basic Challenge Mode (which is the standard mode where the player must survive consecutive challenges until they reach level 100), background theme, graphics, and soundtrack evolve and change together every 4 levels.
The scene is filled with visuals in the background that make the world poppy and alive.
The soundtrack is always one of the protagonists but the player can help boost it even more. Sounds are played when the player rotates the blocks, when they place them, when the Time Line breaks them, or when a score multiplier is activated. The scene can become crowded with sound effects but they all fit the current theme and the game while never being intrusive.
Also, this swarm of sound effects, which the player can somehow control, in conjunction with the soundtrack and the visuals, again allowed Mizuguchi to achieve the feeling of being immersed in the world that he created.
Lumines feels like a precursor to Mizuguchi's next masterpiece: Tetris Effect.
Tetris Effect (2018)
Tetris Effect (TE) stands as Mizuguchi's magnum opus in terms of synesthesia and the fusion of gameplay, visuals, and music. The game is a revamp of Tetris with a newly added mechanic called “the Zone” and additional new modes, including a story mode.
Using the The Zone mechanic the player can slow down time and take full control of the direction and speed of the tetrominoes. Moreover, when The Zone is active, instead of erasing completed lines one by one, each full line is pushed at the bottom, up to a maximum of 16 lines, which will be erased altogether at the end of the phase.
The name of the game itself is taken from an interesting concept: the Tetris Effect occurs when a person devotes so much time and attention to the game that it begins to become an "intoxication", altering the sense of time and space for the player. I am a philosophy enthusiast, and I cannot skip the chance to mention that the Tetris Effect is one of the applications of a philosophical concept called “flow”, defined in 1970 by the psychologist Mihaly Csikszentmihalyi (with whom I almost crossed paths in 2021 while he was still alive and a professor at the University of Chicago).
How Tetris Effect came to be
The story of how TE came to life is a peculiar one. If you have played Lumines, it is easy to see that Mizuguchi had some interest in Tetris. But he could not use the real “Tetris” unless he asked for the use of the license to the Tetris company.
This is where Henk Rogers (one of the two founders of the Tetris company) comes to help. Apparently, Mizuguchi and Henk had known each other for a while and every time they met they used to chat about making a new Tetris game, but without formalizing anything until 2018. At that time VR was starting to take off and Rogers asked Mizuguchi if he would like to make a Tetris game for VR.
Pumped for this new work, Mizuguchi took inspiration from his discussions with Rogers to set some foundational points:
Tetris Effect and Synesthesia
As with all his previous games, synesthesia is everywhere in TE.
Specifically, we have found two different applications of synesthesia in TE: we call them micro- and macro-synesthesia.
The micro one consists of implementing a synesthetic feature into every single tetromino. For example, when playing a level with an underwater setting, each tetromino produces bubbles as if being a living being underwater. Although this feature may seem unnoticeable to the player's eyes, we think it is a perfect fit to the other synesthetic feature in the game and increases the overall impact of synesthesia. Take a look at the GIFs below:
The second application of synesthesia is the macro one. This application includes everything that happens around the Tetris playfield. We think this feature has been much harder to refine and make perfect - because it is right in front of the player at all times.
How did Mizuguchi make something so invasive work in a Tetris game?
Imagine a person driving a car on a long and straight road. Their attention will be focused just ahead, while all the surroundings will be ignored 99% of the time. The same happens when you play Tetris. You focus on the playfield and the tetromino that is falling down, but you barely see anything that happens around the playfield. On the other hand, see below how crowded is the screen when playing TE:
Apparently, when Mizuguchi was testing the game, two opposite issues were happening: some players admitted remembering neither that there were shapes and things going on around the playfield nor that music was playing in the background, while others complained that the images and the sounds were a distraction!
So, how did they reach the final, successful solution? They basically put all the surrounding effects as close as possible to the playfield, but they never added any macro synesthesia effect inside the playfield. This solution allowed them to increase the “heaviness” of the macro synesthesia effect without being too annoying or distracting for the players.
An example might help to understand what we mean here, using the GIF below.
The dolphins could have come "inside" the playfield but they never do. Also, notice how the dolphins are concentrated around the corner of the Tetris playfield - while the extreme border of the screen are much emptier. This is something like placing some obstructions on the side of the road. While you drive you will give a fast glimpse with the corner of the eye to check that you will avoid them, but you don't have to steer the wheel.
In our opinion, this is an interesting solution, which might sound counterintuitive: by excluding macro synesthetic effects from the center stage, you can actually increase exponentially the "weight" of these effects, without any negative impact on the player's experience.
The last point we want to touch on TE is tangentially connected to synesthesia. Continuing on the car metaphor, everyone gets bored after driving a long, straight road for a while, right?
This is the same issue that Mizuguchi faced at this point. The application of synesthesia is cool, but how can you keep players engaged for a long time if they are just playing Tetris?
The solution lies in a two-fold innovation, at game design and gameplay level.
Game Design innovation: Mizuguchi artificially slows down or speeds up the speed of tetrominoes falling down during a match, aligning this pace with the music and the overall synesthetic impact. This is just like having some nice turns while driving, making the whole experience more interesting to the driver. We checked the “turns” in the TE story mode, and this is the graph that shows how often and by how much the falling speed changes when progressing the story mode.
Gameplay innovation: since every one of us is different and might want to take a break at different times during the drive, they added a power up feature called “The Zone”. As said before, The Zone mechanic let the player slow down time and take full control of the direction and speed of the tetrominoes while allowing them to destroy a maximum of 16 lines altogether. With this feature, players are basically able to "take a break" when they need. The music softens immediately when The Zone is activated, enhancing once more the synesthetic connection with the players.
It’s difficult to say if Tetris Effect would have been as fun as it, without the Zone mechanic but this new feature definitely helped to make the game stand-out from the crowd of Tetris-related games.
Conclusions
Although Mizuguchi’s games have had some success and have (probably) influenced many other games (such as Guitar Hero and Hi-Fi Rush), we believe synesthesia is something that should be used more, both in AAA games and especially in VR/AR games, to increase the feeling of immersion and connection with the game. We are seeing more and more applications of the synesthesia concept outside of the gaming industry (have you seen the U2 concert at The Sphere in Las Vegas? Well, that’s exactly what synesthesia is), and we absolutely think there is a lot of opportunity in the gaming industry around this concept.
It has been a while since Mizuguchi has launched his last synesthesia-focused game and we are waiting for a new release to see what they have come up with.
Taking for a moment the player’s perspective on Mizuguchi games, we have loved how he has applied a specific concept, - barely known in the industry in our opinion - and made it his trademark. The biggest challenge yet to be solved is that from time to time playing this kind of game can be overwhelming and tiring if done for a prolonged time.
We tested this by letting our parents play Tetris Effect, they did not last more than 10 minutes!
There are games that we could play for hours and hours, but Mizuguchi’s games are not in this category. Finding some solution to ease to burden on the player’s sensorial experience might be the next step to transforming this genre in something more mainstream and successful with broader audiences.
Anyway, if you have never played any of the games mentioned in this article, we absolutely recommend to try this multisensorial experience at least once!
Thanks for reading, and we look forward to feedback and questions!
Comentarios