AI brings voices back to life. The critical point was "higher harmonics".
//Summary - Level-C2//
In a year-long project, AI was used to regenerate Hibari Misora's singing voice using her extensive audio and video archives. While the technical advances were remarkable, critical feedback highlighted a lack of emotional depth and human warmth in the AI-rendered voice despite its accurate vocal reproduction. The breakthrough came with the identification of Misora's unique ability to produce higher harmonics, a feature central to her expressive singing style, underscoring the challenge of capturing the essence of human artistry through AI.
A)
1)
This is a year-long project to revive Hibari Misora's singing voice using artificial intelligence and AI.
Japan's top engineers developed the AI based on the sound sources and videos left behind by NHK and record companies.
Yasushi Akimoto wrote the lyrics for the new song, the costume design was by Hanae Mori, and the choreography was by Yoshimi Tendo. It was all amazing.
2)
By the way, the photo above is Misora Hibari, which was created in this way.
Finally, we invited the people involved and long-time fans to a concert where we performed only one new song.
3)
To discover how to revive Hibari's singing voice, we analysed a large amount of Hibari Misora sound source data (more than 1,500 songs).
Hibari Misora is said to have a four-octave voice.
All 50 tones, including voiced techniques, are converted into 4-octave data.
In addition, each sound is converted into data by combining elements of vibrato, natural voice, falsetto, and various other expressions.
4)
This is an astronomical task that cannot be done manually by humans.
However, we will overcome this problem using a deep learning system in which the AI repeatedly learns independently.
This way, we will discover the rules for how Hibari Misora sings.
The person in charge of this was a Yamaha engineer who developed Hatsune Miku, a singing voice synthesis software called Vocaloid.
B)
5)
Members of the Misora Hibari Supporters Association listened to the singing voice.
The people who gave the lectures attended concerts during his lifetime, supported him behind the scenes in his later years, and even visited his hospital room.
Their opinions were very harsh.
I can't understand the lyrics with AI's singing voice.
Hibari's singing voice had clear words.
6)
Also, I can't feel the unique power of Hibari's voice.
Hibari's voice makes you feel like you are in a thick atmosphere, but the atmosphere is missing and shallow.
This new song is an actual new song.
Yasushi Akimoto, who wrote the lyrics, felt the same way.
7)
It was inhuman.
We don't want to sing precisely to the score but like a rich, human touch and warmth.
The AI could sing accurately, but it could not impress people.
C)
8)
This is where the analysis of what Misora Hibari possessed that moved people began.
In the programme, we asked a researcher of singing voice analysis to do research.
As a result, I discovered that one of the natural causes was "higher harmonics".
Singers usually produce harmonics up to about 5000Hz, but Misora Hibari could deliver up to 7000Hz.
9)
She gives a little refresher on overtones.
For example, hit the middle C on the piano.
C is 261Hz.
It vibrates 261 times a second.
This is called the fundamental tone.
But in reality, it's not just 261Hz; it's C an octave higher, C two octaves higher, and even higher.
This is the higher harmonics.
10)
Incidentally, the highest note on an 88-key piano is 4186Hz.
This is far below the Misora Hibari's 7000 Hz.
The Misora Hibari doesn't produce many higher harmonics, but only in two or three parts of each song.
11)
In the case of "Like the flow of a river", it was only the "Ka" part of the chorus, "Ah~ Kawa-no Nagare-no Yoni".
He uses different frequencies for each sound.
There is no other singer like her.
That's why she was said to have a mysterious singing voice and a seven-coloured voice, and she could move people to tears and create emotion.
D)
12)
As the analysis progresses, surprising things come to light.
Hibari Misora's songs sound very accurate to the human ear, but when analysed, they are not as accurate or relatively as precise as a machine.
That was one of the things that made me feel human.
E)
13)
It was a new song concert with only one last song, but I cried.
There were some areas where it wasn't smooth and felt a bit artificial, but it was great.
Not just the AI, but Hibari Misora is amazing! That's what I thought.
There is still no one in Japan who can beat Hibari Misora.
By the way, Hibari Misora was also said to be someone who practised extremely until it was perfect.
14)
In conclusion, AI revived Hibari Misora's singing voice, so many people were moved to tears.
So, will the day come when AI will even take on the role of an artist?
Art is said to be the only area where AI has not yet surpassed humans.
Hatsune Miku is also an artist, so humans must work harder.
AI brings voices back to life. The critical point was "higher harmonics".
The digital human with rich reference and the latest AI technology, NHK special "Misora Hibari revived with AI."
https://cgworld.jp/regular/201912-vfxanatomy-cgw256.html
[NHK Special] Misora Hibari Revived with AI | New Song Arekara | NHK
https://www.youtube.com/watch?v=nOLuI7nPQWU
Hibari Misora, an excellent star of the Showa era, is back with AI! AI performs deep learning on the sound sources and videos left by NHK and record companies. She sings a new song "Arekara" produced by Yasushi Akimoto. Singer Yoshimi Tendo collaborated on the choreography. The moment the CG Hibari appeared on stage, the venue felt like it had travelled back in time. "AI Hibari" began to sing after 30 years. Please look at the moving scene, with people crying and shouting "Hibari-chan".
Hibari Misora was a unique entertainer who debuted in the scorched earth of post-war Japan and remained at the top of the music world for a long time. This year, 30 years after her death, we are taking on the challenge of bringing Misora Hibari back into the modern era using the latest AI technology based on the vast number of sound sources and videos left behind by NHK and record companies.
Why did this singing voice touch the hearts of the Japanese people? How can AI quantify and reproduce Hibari Misora's superhuman vocal ability and expressiveness, which until now could only be described in vague terms such as genius or aura?
The new song for AI Hibari was produced by Yasushi Akimoto, who also worked on her last song before her death, ``Kawa no Nagare no Yoni''. Then, analysing a vast amount of past footage, the movements of the eyes and mouth while singing are extracted, and a life-size Hibari Misora appears in a 4K/3D hologram video.
Will Misora Hibari, revived by AI, be able to gain the power to move people's hearts?
