Al needs to be trained to generate dance. Generating dance from music is a challenging task. The main challenges can be listed as below :
l.dance movements need to be aligned well with the given musical style and beats
2.A dance pose at any moments can be followed by various possible movements
3.Long-term structures of body movements results In high kinematic complexity.
MM GAN (Music to Movement GAN) is a technique to create dance movements based on music. in this method, the model first learns how to move by decomposing dance into basic movements. Then it learns how to dance by organizing the basic movements into dance sequences. at the end, it wraps up the dance sequences based on the music beats to generate a long-term dance.
Phase I
1. In Top-Down Decomposition phase the model learns how to move by producing basic movements.
It takes music clip as the input and it tracks the kinematic beat to extract dancing sequences. Then it normalizes the dancing sequences to a series of dance units. At the end of this phase each dance unit is decomposed into an initial pose and possible movements.
Each dance unit consist of a fixed number of poses in a fixed time interval. Dance units are used to capture basic motions patterns and very helpful to find the music style. Also Pose of a current dance unit can be used as the initial pose of the next dance unit to generate a long term dance.
1. In Top-Down Decomposition phase the model learns how to move by producing basic movements.
It takes music clip as the input and it tracks the kinematic beat to extract dancing sequences. Then it normalizes the dancing sequences to a series of dance units. At the end of this phase each dance unit is decomposed into an initial pose and possible movements.
Each dance unit consist of a fixed number of poses in a fixed time interval. Dance units are used to capture basic motions patterns and very helpful to find the music style. Also Pose of a current dance unit can be used as the initial pose of the next dance unit to generate a long term dance.
3. In Testing phase the model wraps the dance units to generate a long-term dance.
From a given input Music first it extracts the style and generates a dance sequence.In the second step it distangels the generated dance to short dance sequences and sample an initial pose randomly. in the third step the model generate a complete dance sequence using the initial pose and short dance sequences ,(the initial pose of the next dance sequence is the taken from the last frame of the current dance sequence). at the end the model wraps the geneared dance sequences by aligning the kinematic beats with the music beat. Figure 6 illustrates the Testing phase procedure.
Figure 2. dance unit examples Experiment
Data collection: Dancing2Music model is trained with 68 K of Ballet , 220 K Zumba and 73 K Hip-hop music clips which equals to 71 hours of training data in total.
Pose Processing :
Dancing2Music applies Open-pose to extract 2D body key-points.in this model 14 key points are chosen as the most relevant key-points to dance process.these key-points are · Nose,neck,left and right shoulders,elbows,wrist,hips,knees, and ankles and the missing key points are estimated by interpolation from the neighbouring frames. Comparison: In our lab, we tested the Dancing2Muisc dance generator to generate a classical dance sequence based on the “Dying Swan” music piece. In parallel, we applied the Open Pose pose detection model on a music video of a ballerina to analyze and compare the Al-generated dance movement and human dance movements from choreographic point of view.
Figure 3.
In ballet all lines are produced by the “en dehors” of the hips, making the feet have a different line and aesthetic. Pointe shoes were created to give ballets an ethereal,volatile and airy atmosphere. For the pose detection model these movements are sometimes indecipherable since the dancer’s foot does not show a common anatomical shape in the dance.
Figure 4.
In this pose the avatar could not deduce that the dancer’s leg “en dehor” is in the “croise” position and interpreted it in a different way, in ballet we could call this pose rather a “duck” and not a swan, referring to a rough pose, whereby even Al needs more training to be able to detect the pose with the “proper” style.
Figure 5.
In this image we see that Al could already distinguish what was the style and position of the legs and arms, even the head and its inclination manage to get quite close to the original.
Semi Humanoid Robots are robots that have the look and feel of a human being or perform an intelligent function of a human being with the support of a human. Semi-humanoid robots are made up of mechanical body parts, intelligent systems (artificial intelligence}, sensors, and human-computer interactions techniques. Pepper is the first social semi-humanoid robot manufactured by SoftBank Robotics, that is able to recognize faces and basic human emotions. Pepper was optimized for human interaction and is able to engage with people through conversation and his touch screen. Pepper’s physical and interactive features raised our interest to design a basic ballet dance sequence for her and experiment human-robot interaction through dance. There are many challenges in designing a choreography for robots including that Pepper doesn’t have legs and has a certain range of movement in each joint which limits her ability to take ballet positions ,However we managed to have robot choreography and design a sequence considering all the limitations. which can be motivation for people with limitations to learn how to dance following a robot.