IT 52 - AIMelody: Cloud-Based Singing Voice Synthesis Web Application

Among the music production professionals, music enthusiasts, hobbyists, or music educators, a good expressive singing voice that can deliver the message of the musician is an important instrument. However, not everyone can sing because they want to. People can be born with a voice that is not suitable for singing, or their voices might be damaged by accidents. Moreover, recording a song from a professional vocalist has a lot of technical challenges, including the need of a professional recording studio, experienced audio engineer, well controlled recording environment, and the performance stability of the vocalist. These requirements often delay the production of audio and music, and high entry barriers lower the interest of people in music.

On the other hand, there always exists a need to create a customized synthetic singing voice to allow users to create singing vocals of their own. Situations include:

• Songwriting and music production: Composers employ synthetic singing to quickly test and refine their musical compositions and can use synthetic voices for unique vocal effects.
• Accessibility for amateurs, hobbyists, or music educators: They can utilize synthetic voices to create music, overcoming limitations in singing ability.
• Multimedia and Gaming: Multimedia and gaming industries use synthetic singing for character voices and dynamic soundtracks.
• Language Learning and Cultural Exploration: Language learning platforms incorporate synthetic singing to enhance language practice through music.

The creation of a synthetic singing voice involves a combination of audio analysis and machine learning algorithms. It starts with analyzing lyrics into phonemes, which are then mapped onto a musical score. Each phoneme is synthesized to match the pitch, duration, and synthesized into melody. Advanced algorithms and machine learning optimize the synthesis and adapt the voice to various languages and styles.

Despite the difficulties, there are a few solutions in the industry (e.g. Yamaha Vocaloid, Dreamtonics Synthesizer, UTAU). However, these products have common limitations. For example, all of them are desktop-based solutions, or the solutions exist in only few languages (predominantly Japanese, Mandarin Chinese, and English), and lastly, all of them charges purchase fees. 

We are a group of music enthusiasts, software developers, and artificial intelligence practitioners. Our goal for this project is to create a handy, easy-to-use application that helps with the above business needs. Our proposed solution is a cloud-based web application that integrates with existing singing voice synthesis (SVS) engines and allows users to provide a desired lyrics and music score, and the application will generate a synthetic singing voices from the samples.

Our AIMelody app makes it easy for anyone to create their own singing voice without needing expensive equipment or special skills. The application works with internet access, and users can use it from anywhere. Professional users can experiment with different vocal styles and languages, enhancing creativity in songwriting and music production. For amateurs and educators, it offers a tool to overcome singing limitations and engage in music creation. It lowers the barrier for people to connect with music and singing. AIMelody aims to democratize the use of synthetic singing, making singing easy for everyone.